2024-03-29T10:44:06.085 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T10:44:06.085 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/orig.config.yaml 2024-03-29T10:44:06.121 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T10:44:06.209 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622663' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: zstd bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyACxNVF/D20cEsz4I+IiCBGhDOKyJdc1ozs7gI/OY9v8Yi188AgmNl4XqE171C391oJoQ0WCjrZbTwX7R5NnY= smithi026.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwmYzMU7GD+l5piL0UNy8nu1PbS3hfMr10PIsafCqxd5VMKzXTWrdcyK3b1TFXRWHs5lh+mJVBBao2kK5ilz54= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T10:44:06.210 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T10:44:06.210 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T10:44:06.211 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T10:44:06.212 INFO:teuthology.task.internal:Checking packages... 2024-03-29T10:44:06.236 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T10:44:06.237 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T10:44:06.237 INFO:teuthology.packaging:ref: None 2024-03-29T10:44:06.237 INFO:teuthology.packaging:tag: None 2024-03-29T10:44:06.237 INFO:teuthology.packaging:branch: main 2024-03-29T10:44:06.237 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:44:06.237 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T10:44:06.637 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T10:44:06.639 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T10:44:06.646 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T10:44:06.646 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T10:44:06.655 INFO:teuthology.task.internal:Saving configuration 2024-03-29T10:44:06.668 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T10:44:06.674 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T10:44:06.698 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:39:22.272230', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyACxNVF/D20cEsz4I+IiCBGhDOKyJdc1ozs7gI/OY9v8Yi188AgmNl4XqE171C391oJoQ0WCjrZbTwX7R5NnY='} 2024-03-29T10:44:06.718 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:39:22.273777', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwmYzMU7GD+l5piL0UNy8nu1PbS3hfMr10PIsafCqxd5VMKzXTWrdcyK3b1TFXRWHs5lh+mJVBBao2kK5ilz54='} 2024-03-29T10:44:06.718 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T10:44:06.724 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T10:44:06.724 INFO:teuthology.task.internal:roles: ubuntu@smithi026.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T10:44:06.724 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T10:44:06.796 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd01262c0d0>, signals=[15]) 2024-03-29T10:44:06.796 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T10:44:06.802 INFO:teuthology.task.internal:Opening connections... 2024-03-29T10:44:06.802 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T10:44:06.804 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:06.910 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T10:44:06.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:44:07.012 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T10:44:07.019 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2024-03-29T10:44:07.038 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2024-03-29T10:44:07.038 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="9" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="9" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2024-03-29T10:44:07.096 INFO:teuthology.orchestra.run.smithi022.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:44:07.097 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:44:07.097 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:44:07.097 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:44:07.097 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:44:07.097 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:44:07.097 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2024-03-29T10:44:07.128 DEBUG:teuthology.orchestra.run.smithi026:> uname -m 2024-03-29T10:44:07.146 INFO:teuthology.orchestra.run.smithi026.stdout:x86_64 2024-03-29T10:44:07.147 DEBUG:teuthology.orchestra.run.smithi026:> cat /etc/os-release 2024-03-29T10:44:07.204 INFO:teuthology.orchestra.run.smithi026.stdout:NAME="CentOS Stream" 2024-03-29T10:44:07.204 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION="9" 2024-03-29T10:44:07.204 INFO:teuthology.orchestra.run.smithi026.stdout:ID="centos" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:ID_LIKE="rhel fedora" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:VERSION_ID="9" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:PLATFORM_ID="platform:el9" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:ANSI_COLOR="0;31" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:LOGO="fedora-logo-icon" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:HOME_URL="https://centos.org/" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T10:44:07.205 INFO:teuthology.orchestra.run.smithi026.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T10:44:07.205 INFO:teuthology.lock.ops:Updating smithi026.front.sepia.ceph.com on lock server 2024-03-29T10:44:07.228 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T10:44:07.237 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T10:44:07.243 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T10:44:07.243 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:44:07.246 DEBUG:teuthology.orchestra.run.smithi026:> test '!' -e /home/ubuntu/cephtest 2024-03-29T10:44:07.262 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T10:44:07.269 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T10:44:07.269 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:44:07.303 DEBUG:teuthology.orchestra.run.smithi026:> test -z $(ls -A /var/lib/ceph) 2024-03-29T10:44:07.326 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T10:44:07.373 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T10:44:07.387 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T10:44:07.387 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T10:44:07.387 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:44:07.388 DEBUG:teuthology.orchestra.run.smithi026:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T10:44:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:44:07.405 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-29T10:44:07.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:44:07.406 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-29T10:44:07.463 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:44:07.463 INFO:teuthology.task.kernel:Running kernel on smithi022: 5.14.0-361.el9.x86_64 2024-03-29T10:44:07.464 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2024-03-29T10:44:07.465 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-361.el9.x86_64 2024-03-29T10:44:07.465 INFO:teuthology.task.kernel:Running kernel on smithi026: 5.14.0-361.el9.x86_64 2024-03-29T10:44:07.465 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum install -y kernel 2024-03-29T10:44:09.601 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-29T10:44:09.742 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 10 MB/s | 6.7 MB 00:00 2024-03-29T10:44:11.126 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-29T10:44:11.183 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-29T10:44:12.435 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 8.9 MB/s | 8.0 MB 00:00 2024-03-29T10:44:12.491 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-03-29T10:44:14.350 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-29T10:44:15.050 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-29T10:44:18.795 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 8.2 MB/s | 6.1 MB 00:00 2024-03-29T10:44:19.367 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 8.5 MB/s | 6.1 MB 00:00 2024-03-29T10:44:20.194 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 73 kB/s | 16 kB 00:00 2024-03-29T10:44:21.035 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 32 kB/s | 16 kB 00:00 2024-03-29T10:44:21.572 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-29T10:44:22.719 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-29T10:44:26.728 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-03-29T10:44:27.878 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T10:44:29.093 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:44:29.094 INFO:teuthology.orchestra.run.smithi026.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:44:29.188 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:44:29.198 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:44:29.198 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:44:29.198 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:44:29.198 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:44:29.198 INFO:teuthology.orchestra.run.smithi026.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:44:29.198 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout:Install 4 Packages 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 76 M 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 123 M 2024-03-29T10:44:29.199 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:44:29.380 INFO:teuthology.orchestra.run.smithi026.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.0 MB/s | 192 kB 00:00 2024-03-29T10:44:30.038 INFO:teuthology.orchestra.run.smithi026.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-29T10:44:30.268 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T10:44:30.268 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T10:44:30.363 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:44:30.374 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:44:30.374 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout:Install 4 Packages 2024-03-29T10:44:30.375 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:44:30.376 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 76 M 2024-03-29T10:44:30.376 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 123 M 2024-03-29T10:44:30.376 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:44:30.996 INFO:teuthology.orchestra.run.smithi022.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 309 kB/s | 192 kB 00:00 2024-03-29T10:44:31.163 INFO:teuthology.orchestra.run.smithi026.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 17 MB/s | 34 MB 00:01 2024-03-29T10:44:31.838 INFO:teuthology.orchestra.run.smithi022.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 10 MB/s | 15 MB 00:01 2024-03-29T10:44:32.513 INFO:teuthology.orchestra.run.smithi022.stdout:(3/4): kernel-modules-core-5.14.0-434.el9.x86_6 18 MB/s | 28 MB 00:01 2024-03-29T10:44:32.738 INFO:teuthology.orchestra.run.smithi026.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 8.2 MB/s | 28 MB 00:03 2024-03-29T10:44:32.741 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:32.742 INFO:teuthology.orchestra.run.smithi026.stdout:Total 22 MB/s | 76 MB 00:03 2024-03-29T10:44:32.742 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:44:32.813 INFO:teuthology.orchestra.run.smithi022.stdout:(4/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 14 MB/s | 34 MB 00:02 2024-03-29T10:44:32.816 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:44:32.816 INFO:teuthology.orchestra.run.smithi022.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-29T10:44:32.816 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:44:33.186 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:44:33.186 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:44:33.265 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:44:33.265 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:44:34.090 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:44:34.091 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:44:34.138 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:44:34.138 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:44:36.527 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:44:36.598 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:44:37.135 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:44:37.213 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:44:37.228 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:44:37.314 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:44:40.122 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:44:40.197 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:44:40.256 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:44:40.333 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:44:45.328 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:44:45.380 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:44:45.414 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:44:45.459 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:44:50.484 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:44:50.556 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:21.112 INFO:teuthology.orchestra.run.smithi022.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:45:21.112 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:45:21.158 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:21.371 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:21.371 INFO:teuthology.orchestra.run.smithi026.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T10:45:21.371 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:45:21.415 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:21.544 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:21.544 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:45:21.544 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:45:21.544 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:45:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:21.780 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T10:45:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T10:45:21.781 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T10:45:22.038 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:22.038 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:45:22.039 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:45:22.039 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.039 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.039 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.039 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.039 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:45:22.039 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:45:22.186 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T10:45:22.186 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:45:22.186 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:45:22.186 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.186 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.186 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.187 INFO:teuthology.orchestra.run.smithi026.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.187 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:45:22.187 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:45:22.388 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:45:22.489 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.489 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.489 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.489 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:45:22.490 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:45:22.490 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T10:45:22.490 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:45:22.490 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:45:22.512 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:45:22.573 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.573 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.573 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2024-03-29T10:45:22.617 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2024-03-29T10:45:22.617 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2024-03-29T10:45:22.617 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.06 2024-03-29T10:45:22.617 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 68.el9 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Group : Unspecified 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 8264254 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : builder@centos.org 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : CentOS 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:45:22.618 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:45:22.619 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:45:22.620 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.621 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:45:22.621 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.622 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:45:22.623 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.623 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.623 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T10:45:22.623 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T10:45:22.623 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T10:45:22.623 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi026.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T10:45:22.623 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T10:45:22.708 INFO:teuthology.orchestra.run.smithi026.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T10:45:22.709 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.710 DEBUG:teuthology.orchestra.run.smithi026:> sudo rpm -qi grub2-tools 2024-03-29T10:45:22.754 INFO:teuthology.orchestra.run.smithi026.stdout:Name : grub2-tools 2024-03-29T10:45:22.754 INFO:teuthology.orchestra.run.smithi026.stdout:Epoch : 1 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Version : 2.06 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Release : 68.el9 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Architecture: x86_64 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Group : Unspecified 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Size : 8264254 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:License : GPLv3+ 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Packager : builder@centos.org 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:Vendor : CentOS 2024-03-29T10:45:22.755 INFO:teuthology.orchestra.run.smithi026.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout:Summary : Support tools for GRUB. 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout:Description : 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout:hardware devices. 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:45:22.756 INFO:teuthology.orchestra.run.smithi026.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T10:45:22.757 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T10:45:22.757 INFO:teuthology.task.kernel:Updating grub on smithi026 to boot 5.14.0-434.el9.x86_64 2024-03-29T10:45:22.757 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T10:45:23.172 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2024-03-29T10:45:23.338 INFO:teuthology.orchestra.run.smithi026.stderr:Generating grub configuration file ... 2024-03-29T10:45:24.586 INFO:teuthology.orchestra.run.smithi022.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:45:24.616 INFO:teuthology.orchestra.run.smithi022.stderr:done 2024-03-29T10:45:24.619 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-29T10:45:24.634 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.bv7sLNRZB5 2024-03-29T10:45:24.634 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.bv7sLNRZB5 2024-03-29T10:45:24.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.bv7sLNRZB5 2024-03-29T10:45:24.779 INFO:teuthology.orchestra.run.smithi026.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T10:45:24.816 INFO:teuthology.orchestra.run.smithi026.stderr:done 2024-03-29T10:45:24.819 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-29T10:45:24.834 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.Q5FbwyifmR 2024-03-29T10:45:24.834 DEBUG:teuthology.orchestra.run.smithi026:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Q5FbwyifmR 2024-03-29T10:45:24.855 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.bv7sLNRZB5 is 6KB 2024-03-29T10:45:24.904 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.bv7sLNRZB5 2024-03-29T10:45:24.908 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.Q5FbwyifmR 2024-03-29T10:45:24.918 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:45:24.986 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:45:24.986 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:45:24.987 INFO:teuthology.orchestra.run.smithi022.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:45:24.988 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:45:25.054 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.Q5FbwyifmR is 6KB 2024-03-29T10:45:25.103 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.Q5FbwyifmR 2024-03-29T10:45:25.118 DEBUG:teuthology.orchestra.run.smithi026:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T10:45:25.149 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2024-03-29T10:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T10:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T10:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T10:45:25.186 INFO:teuthology.orchestra.run.smithi026.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T10:45:25.187 DEBUG:teuthology.orchestra.run.smithi026:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T10:45:25.346 DEBUG:teuthology.orchestra.run.smithi026:> sudo shutdown -r now 2024-03-29T10:45:55.178 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T10:45:55.179 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T10:45:55.180 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:45:55.349 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T10:45:55.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-29T10:45:55.349 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:46:13.534 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.22 2024-03-29T10:46:13.790 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-29T10:46:22.544 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T10:46:22.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:46:22.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-29T10:46:22.792 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:46:41.186 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.26 2024-03-29T10:46:53.198 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-29T10:46:53.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:46:53.707 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-29T10:46:54.089 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-29T10:46:54.089 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:46:54.089 DEBUG:teuthology.orchestra.run.smithi026:> uname -r 2024-03-29T10:46:54.103 INFO:teuthology.orchestra.run.smithi026.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:46:54.103 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:46:54.103 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:46:54.103 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:46:54.434 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-29T10:46:54.868 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T10:46:54.868 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T10:46:54.868 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2024-03-29T10:46:54.883 INFO:teuthology.orchestra.run.smithi022.stdout:5.14.0-434.el9.x86_64 2024-03-29T10:46:54.883 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T10:46:54.883 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T10:46:54.883 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T10:46:55.104 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T10:46:55.104 DEBUG:teuthology.orchestra.run.smithi026:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:46:55.197 INFO:teuthology.orchestra.run.smithi026.stdout:ttyS1 2024-03-29T10:46:55.238 DEBUG:teuthology.parallel:result is None 2024-03-29T10:46:55.884 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T10:46:55.885 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T10:46:55.973 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2024-03-29T10:46:56.020 DEBUG:teuthology.parallel:result is None 2024-03-29T10:46:56.020 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T10:46:56.027 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T10:46:56.027 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:46:56.030 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T10:46:56.045 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T10:46:56.075 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T10:46:56.084 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T10:46:56.084 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:46:56.088 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T10:46:56.126 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T10:46:56.133 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T10:46:56.134 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:46:56.164 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T10:46:56.191 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:46:56.202 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:46:56.204 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:46:56.216 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T10:46:56.217 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T10:46:56.224 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T10:46:56.225 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:46:56.261 DEBUG:teuthology.orchestra.run.smithi026:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T10:46:56.288 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T10:46:56.297 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T10:46:56.298 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:46:56.330 DEBUG:teuthology.orchestra.run.smithi026:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T10:46:56.343 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:46:56.430 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:46:56.470 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:46:56.551 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:46:56.595 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:46:56.596 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:46:56.687 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:46:56.728 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T10:46:56.777 DEBUG:teuthology.orchestra.run.smithi026:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:46:56.855 DEBUG:teuthology.orchestra.run.smithi026:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T10:46:56.899 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:46:56.899 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T10:46:56.965 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2024-03-29T10:46:56.967 DEBUG:teuthology.orchestra.run.smithi026:> sudo service rsyslog restart 2024-03-29T10:46:57.007 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:46:57.071 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T10:46:57.387 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T10:46:57.395 INFO:teuthology.task.internal:Starting timer... 2024-03-29T10:46:57.395 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T10:46:57.406 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T10:46:57.414 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2024-03-29T10:46:57.468 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: 2024-03-29T10:46:57.470 DEBUG:teuthology.orchestra.run.smithi026:> sudo service auditd rotate 2024-03-29T10:46:57.564 INFO:teuthology.orchestra.run.smithi026.stdout:Rotating logs: 2024-03-29T10:46:57.567 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T10:46:57.567 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2024-03-29T10:46:57.587 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2024-03-29T10:46:57.587 DEBUG:teuthology.orchestra.run.smithi026:> /usr/sbin/getenforce 2024-03-29T10:46:57.618 INFO:teuthology.orchestra.run.smithi026.stdout:Permissive 2024-03-29T10:46:57.619 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi022.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi026.front.sepia.ceph.com': 'permissive'} 2024-03-29T10:46:57.619 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:46:57.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:46:57.655 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T10:46:57.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T10:46:57.711 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T10:46:57.711 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:46:57.746 DEBUG:teuthology.orchestra.run.smithi026:> sudo /usr/sbin/setenforce permissive 2024-03-29T10:46:57.971 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T10:46:57.981 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-29T10:46:57.989 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-29T10:46:58.118 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T10:46:58.135 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T10:46:58.136 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi022.front.sepia.ceph.com,smithi026.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T10:51:19.189 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi022.front.sepia.ceph.com'), Remote(name='ubuntu@smithi026.front.sepia.ceph.com')] 2024-03-29T10:51:19.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T10:51:19.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:51:19.287 DEBUG:teuthology.orchestra.run.smithi022:> true 2024-03-29T10:51:19.367 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2024-03-29T10:51:19.368 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-29T10:51:19.369 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi026.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T10:51:19.505 DEBUG:teuthology.orchestra.run.smithi026:> true 2024-03-29T10:51:19.595 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi026.front.sepia.ceph.com' 2024-03-29T10:51:19.595 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T10:51:19.615 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T10:51:19.615 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:51:19.616 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:51:19.618 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T10:51:19.619 DEBUG:teuthology.orchestra.run.smithi026:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T10:51:19.648 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:51:19.666 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:51:19.667 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T10:51:19.685 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T10:51:19.696 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2024-03-29T10:51:19.710 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2024-03-29T10:51:19.715 INFO:teuthology.orchestra.run.smithi026.stderr:sudo: ntpd: command not found 2024-03-29T10:51:19.726 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:51:19.729 INFO:teuthology.orchestra.run.smithi026.stdout:506 Cannot talk to daemon 2024-03-29T10:51:19.743 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:51:19.745 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T10:51:19.762 INFO:teuthology.orchestra.run.smithi026.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T10:51:19.796 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:51:19.799 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:51:19.799 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-29T10:51:19.799 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.799 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.799 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.799 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.816 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-29T10:51:19.819 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T10:51:19.819 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-29T10:51:19.819 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.819 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.820 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.820 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T10:51:19.821 INFO:teuthology.run_tasks:Running task install... 2024-03-29T10:51:19.831 DEBUG:teuthology.task.install:project ceph 2024-03-29T10:51:19.831 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T10:51:19.831 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T10:51:19.831 INFO:teuthology.task.install:Using flavor: default 2024-03-29T10:51:19.837 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T10:51:19.837 INFO:teuthology.task.install:extra packages: [] 2024-03-29T10:51:19.837 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:51:19.838 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:51:19.840 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T10:51:19.840 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:51:19.986 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:51:19.986 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:51:19.990 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T10:51:19.990 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T10:51:20.101 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:51:20.102 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:51:20.102 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:51:20.104 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T10:51:20.105 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:51:20.105 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T10:51:20.129 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:51:20.130 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:51:20.134 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T10:51:20.134 DEBUG:teuthology.orchestra.run.smithi026:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T10:51:20.198 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:51:20.201 DEBUG:teuthology.orchestra.run.smithi026:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T10:51:20.278 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:51:20.280 DEBUG:teuthology.orchestra.run.smithi026:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T10:51:20.344 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2024-03-29T10:51:20.345 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2024-03-29T10:51:20.346 INFO:teuthology.orchestra.run.smithi026.stdout:check_obsoletes = 1 2024-03-29T10:51:20.347 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum clean all 2024-03-29T10:51:20.586 INFO:teuthology.orchestra.run.smithi022.stdout:74 files removed 2024-03-29T10:51:20.602 INFO:teuthology.orchestra.run.smithi026.stdout:74 files removed 2024-03-29T10:51:20.613 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw 2024-03-29T10:51:20.629 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-radosgw 2024-03-29T10:51:21.413 INFO:teuthology.orchestra.run.smithi026.stdout:ceph packages for x86_64 171 kB/s | 82 kB 00:00 2024-03-29T10:51:21.476 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 147 kB/s | 82 kB 00:00 2024-03-29T10:51:21.946 INFO:teuthology.orchestra.run.smithi026.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-29T10:51:22.010 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 21 kB/s | 10 kB 00:00 2024-03-29T10:51:22.281 INFO:teuthology.orchestra.run.smithi026.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2024-03-29T10:51:22.335 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-03-29T10:51:22.933 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-29T10:51:23.389 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - AppStream 6.6 MB/s | 6.7 MB 00:01 2024-03-29T10:51:24.408 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-29T10:51:24.889 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS-9 - BaseOS 6.4 MB/s | 2.0 MB 00:00 2024-03-29T10:51:25.533 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-03-29T10:51:27.753 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-03-29T10:51:28.844 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 19 MB 00:02 2024-03-29T10:51:30.163 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-29T10:51:40.583 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - CRB 772 kB/s | 6.1 MB 00:08 2024-03-29T10:51:41.587 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - CRB 804 kB/s | 6.1 MB 00:07 2024-03-29T10:51:42.067 INFO:teuthology.orchestra.run.smithi026.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-03-29T10:51:43.105 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-03-29T10:51:43.362 INFO:teuthology.orchestra.run.smithi026.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-29T10:51:45.259 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-29T10:51:48.479 INFO:teuthology.orchestra.run.smithi026.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-29T10:51:50.414 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 44 kB/s | 1.7 kB 00:00 2024-03-29T10:51:51.171 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrading: 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:51:51.174 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:51:51.175 INFO:teuthology.orchestra.run.smithi026.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:51:51.176 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout:Installing weak dependencies: 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout:Install 30 Packages 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout:Upgrade 2 Packages 2024-03-29T10:51:51.177 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:51:51.178 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 58 M 2024-03-29T10:51:51.178 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:51:51.947 INFO:teuthology.orchestra.run.smithi026.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 8.8 MB/s | 5.3 MB 00:00 2024-03-29T10:51:52.005 INFO:teuthology.orchestra.run.smithi026.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 25 kB 00:00 2024-03-29T10:51:52.097 INFO:teuthology.orchestra.run.smithi026.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-29T10:51:52.181 INFO:teuthology.orchestra.run.smithi026.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 5.8 MB/s | 498 kB 00:00 2024-03-29T10:51:52.639 INFO:teuthology.orchestra.run.smithi026.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 11 MB/s | 5.1 MB 00:00 2024-03-29T10:51:52.715 INFO:teuthology.orchestra.run.smithi026.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 602 kB/s | 45 kB 00:00 2024-03-29T10:51:52.773 INFO:teuthology.orchestra.run.smithi026.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 2.2 MB/s | 131 kB 00:00 2024-03-29T10:51:52.965 INFO:teuthology.orchestra.run.smithi026.stdout:(8/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 12 MB/s | 20 MB 00:01 2024-03-29T10:51:52.991 INFO:teuthology.orchestra.run.smithi026.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 746 kB/s | 162 kB 00:00 2024-03-29T10:51:53.016 INFO:teuthology.orchestra.run.smithi026.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 6.3 MB/s | 320 kB 00:00 2024-03-29T10:51:53.041 INFO:teuthology.orchestra.run.smithi026.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 5.8 MB/s | 298 kB 00:00 2024-03-29T10:51:53.066 INFO:teuthology.orchestra.run.smithi026.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.9 MB/s | 99 kB 00:00 2024-03-29T10:51:53.129 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:51:53.142 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T10:51:53.143 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T10:51:53.144 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout:Installing weak dependencies: 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout:Install 30 Packages 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 2 Packages 2024-03-29T10:51:53.145 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:51:53.146 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 58 M 2024-03-29T10:51:53.146 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:51:53.185 INFO:teuthology.orchestra.run.smithi026.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 737 kB/s | 106 kB 00:00 2024-03-29T10:51:53.217 INFO:teuthology.orchestra.run.smithi026.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-29T10:51:53.234 INFO:teuthology.orchestra.run.smithi026.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 910 kB/s | 44 kB 00:00 2024-03-29T10:51:53.284 INFO:teuthology.orchestra.run.smithi026.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.6 MB/s | 661 kB 00:00 2024-03-29T10:51:53.309 INFO:teuthology.orchestra.run.smithi026.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-29T10:51:53.326 INFO:teuthology.orchestra.run.smithi026.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 997 kB/s | 41 kB 00:00 2024-03-29T10:51:53.351 INFO:teuthology.orchestra.run.smithi026.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 5.1 MB/s | 220 kB 00:00 2024-03-29T10:51:53.368 INFO:teuthology.orchestra.run.smithi026.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 835 kB/s | 35 kB 00:00 2024-03-29T10:51:53.385 INFO:teuthology.orchestra.run.smithi026.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.3 MB/s | 46 kB 00:00 2024-03-29T10:51:53.620 INFO:teuthology.orchestra.run.smithi026.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-03-29T10:51:53.662 INFO:teuthology.orchestra.run.smithi026.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 626 kB/s | 26 kB 00:00 2024-03-29T10:51:53.711 INFO:teuthology.orchestra.run.smithi026.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 976 kB/s | 49 kB 00:00 2024-03-29T10:51:53.770 INFO:teuthology.orchestra.run.smithi026.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T10:51:53.828 INFO:teuthology.orchestra.run.smithi026.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 10 MB/s | 4.4 MB 00:00 2024-03-29T10:51:53.889 INFO:teuthology.orchestra.run.smithi026.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-29T10:51:53.921 INFO:teuthology.orchestra.run.smithi026.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.4 MB/s | 838 kB 00:00 2024-03-29T10:51:53.996 INFO:teuthology.orchestra.run.smithi026.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-29T10:51:54.838 INFO:teuthology.orchestra.run.smithi026.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 3.7 MB/s | 3.4 MB 00:00 2024-03-29T10:51:54.882 INFO:teuthology.orchestra.run.smithi022.stdout:(1/32): ceph-common-19.0.0-1439.g5dcb190d.el9.x 13 MB/s | 20 MB 00:01 2024-03-29T10:51:54.926 INFO:teuthology.orchestra.run.smithi022.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 610 kB/s | 25 kB 00:00 2024-03-29T10:51:54.971 INFO:teuthology.orchestra.run.smithi026.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 3.2 MB/s | 3.1 MB 00:00 2024-03-29T10:51:55.016 INFO:teuthology.orchestra.run.smithi022.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 7.9 MB/s | 742 kB 00:00 2024-03-29T10:51:55.082 INFO:teuthology.orchestra.run.smithi022.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 7.3 MB/s | 498 kB 00:00 2024-03-29T10:51:55.266 INFO:teuthology.orchestra.run.smithi022.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 2.8 MB/s | 5.3 MB 00:01 2024-03-29T10:51:55.358 INFO:teuthology.orchestra.run.smithi022.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 492 kB/s | 45 kB 00:00 2024-03-29T10:51:55.425 INFO:teuthology.orchestra.run.smithi022.stdout:(7/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 15 MB/s | 5.1 MB 00:00 2024-03-29T10:51:55.450 INFO:teuthology.orchestra.run.smithi022.stdout:(8/32): python3-ceph-common-19.0.0-1439.g5dcb19 1.4 MB/s | 131 kB 00:00 2024-03-29T10:51:55.476 INFO:teuthology.orchestra.run.smithi022.stdout:(9/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 3.2 MB/s | 162 kB 00:00 2024-03-29T10:51:55.518 INFO:teuthology.orchestra.run.smithi022.stdout:(10/32): python3-rados-19.0.0-1439.g5dcb190d.el 4.7 MB/s | 320 kB 00:00 2024-03-29T10:51:55.521 INFO:teuthology.orchestra.run.smithi026.stdout:(32/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 2.3 MB/s | 9.7 MB 00:04 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi026.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-29T10:51:55.527 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:51:55.543 INFO:teuthology.orchestra.run.smithi022.stdout:(11/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 4.3 MB/s | 298 kB 00:00 2024-03-29T10:51:55.569 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:51:55.570 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:51:55.601 INFO:teuthology.orchestra.run.smithi022.stdout:(12/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 1.2 MB/s | 99 kB 00:00 2024-03-29T10:51:55.660 INFO:teuthology.orchestra.run.smithi022.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 907 kB/s | 106 kB 00:00 2024-03-29T10:51:55.685 INFO:teuthology.orchestra.run.smithi022.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-29T10:51:55.702 INFO:teuthology.orchestra.run.smithi022.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T10:51:55.811 INFO:teuthology.orchestra.run.smithi022.stdout:(16/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 3.9 MB/s | 9.7 MB 00:02 2024-03-29T10:51:55.836 INFO:teuthology.orchestra.run.smithi022.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 661 kB 00:00 2024-03-29T10:51:55.861 INFO:teuthology.orchestra.run.smithi022.stdout:(18/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-29T10:51:55.887 INFO:teuthology.orchestra.run.smithi022.stdout:(19/32): python3-prettytable-0.7.2-27.el9.noarc 557 kB/s | 41 kB 00:00 2024-03-29T10:51:55.912 INFO:teuthology.orchestra.run.smithi022.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T10:51:55.954 INFO:teuthology.orchestra.run.smithi022.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 380 kB/s | 35 kB 00:00 2024-03-29T10:51:55.979 INFO:teuthology.orchestra.run.smithi022.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 506 kB/s | 46 kB 00:00 2024-03-29T10:51:56.054 INFO:teuthology.orchestra.run.smithi022.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 347 kB/s | 26 kB 00:00 2024-03-29T10:51:56.055 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:51:56.055 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:51:56.129 INFO:teuthology.orchestra.run.smithi022.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 648 kB/s | 49 kB 00:00 2024-03-29T10:51:56.155 INFO:teuthology.orchestra.run.smithi022.stdout:(25/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.5 MB/s | 309 kB 00:00 2024-03-29T10:51:56.272 INFO:teuthology.orchestra.run.smithi022.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 474 kB/s | 67 kB 00:00 2024-03-29T10:51:56.339 INFO:teuthology.orchestra.run.smithi022.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T10:51:56.406 INFO:teuthology.orchestra.run.smithi022.stdout:(28/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-29T10:51:56.540 INFO:teuthology.orchestra.run.smithi022.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 4.4 MB 00:00 2024-03-29T10:51:56.581 INFO:teuthology.orchestra.run.smithi022.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-29T10:51:56.815 INFO:teuthology.orchestra.run.smithi022.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 3.1 MB 00:00 2024-03-29T10:51:57.082 INFO:teuthology.orchestra.run.smithi022.stdout:(32/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 5.0 MB/s | 3.4 MB 00:00 2024-03-29T10:51:57.086 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:51:57.086 INFO:teuthology.orchestra.run.smithi022.stdout:Total 15 MB/s | 58 MB 00:03 2024-03-29T10:51:57.086 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:51:57.129 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:51:57.129 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:51:57.536 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:51:57.626 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:51:57.626 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:51:57.688 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:51:57.794 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:51:57.819 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:51:57.851 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:51:57.866 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:51:58.060 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:51:58.115 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:51:58.456 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:51:58.503 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:51:58.601 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:51:58.685 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:51:58.713 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:51:58.775 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:51:58.796 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:51:58.833 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:51:58.902 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:51:58.924 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:51:58.987 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:51:59.022 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:51:59.132 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:51:59.177 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:51:59.196 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:51:59.214 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:51:59.247 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:51:59.262 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T10:51:59.276 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:51:59.308 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:51:59.314 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T10:51:59.345 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T10:51:59.368 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:51:59.385 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T10:51:59.576 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:51:59.642 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T10:51:59.643 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:51:59.671 INFO:teuthology.orchestra.run.smithi026.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:51:59.745 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:51:59.776 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:51:59.817 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:51:59.984 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T10:52:00.029 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:52:00.122 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:52:00.212 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T10:52:00.239 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:52:00.288 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T10:52:00.294 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:52:00.323 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T10:52:00.323 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:52:00.359 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T10:52:00.386 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:52:00.428 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:52:00.450 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:52:00.457 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:52:00.484 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:52:00.514 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T10:52:00.549 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T10:52:00.662 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T10:52:00.700 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T10:52:00.763 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T10:52:00.797 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T10:52:00.826 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T10:52:00.859 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T10:52:01.191 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T10:52:01.231 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:52:01.313 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T10:52:01.352 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T10:52:01.396 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T10:52:01.876 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T10:52:01.908 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:52:01.980 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T10:52:02.051 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T10:52:02.085 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T10:52:02.734 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:52:02.772 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:52:03.374 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:52:03.409 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:52:03.496 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:52:03.497 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:52:03.497 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:52:03.497 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:03.535 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:52:04.306 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:52:04.358 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:52:04.940 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T10:52:04.976 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:52:05.061 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T10:52:05.061 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T10:52:05.061 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T10:52:05.061 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:05.094 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /sys 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /proc 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /mnt 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /var/tmp 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /home 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /root 2024-03-29T10:52:13.812 INFO:teuthology.orchestra.run.smithi026.stdout:skipping the directory /tmp 2024-03-29T10:52:13.813 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:14.732 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:52:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:52:14.756 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:14.756 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:52:14.756 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:52:14.756 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:52:14.756 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:14.758 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:52:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:52:14.842 INFO:teuthology.orchestra.run.smithi026.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:52:15.190 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:52:15.190 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2024-03-29T10:52:15.191 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2024-03-29T10:52:15.191 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2024-03-29T10:52:15.191 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2024-03-29T10:52:15.191 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2024-03-29T10:52:15.191 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /root 2024-03-29T10:52:15.191 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2024-03-29T10:52:15.191 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:52:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:52:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:52:15.952 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:52:15.953 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:52:15.954 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:52:15.955 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:52:16.129 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:52:16.154 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T10:52:16.154 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:16.154 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T10:52:16.154 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:52:16.154 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T10:52:16.155 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:16.159 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:52:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T10:52:16.240 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:52:16.975 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:52:16.975 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout:Upgraded: 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.976 INFO:teuthology.orchestra.run.smithi026.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.977 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:16.978 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T10:52:17.313 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T10:52:17.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T10:52:17.315 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T10:52:17.327 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-test 2024-03-29T10:52:17.894 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:18.051 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:18.052 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:52:18.052 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:18.052 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:52:18.052 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:52:18.052 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:52:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:52:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:52:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:52:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:52:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:52:18.053 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:18.054 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:52:18.054 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:18.054 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-03-29T10:52:18.054 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:18.055 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 48 M 2024-03-29T10:52:18.055 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 204 M 2024-03-29T10:52:18.055 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:52:18.314 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T10:52:18.314 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:18.314 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2024-03-29T10:52:18.314 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T10:52:18.315 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T10:52:18.316 INFO:teuthology.orchestra.run.smithi022.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T10:52:18.317 INFO:teuthology.orchestra.run.smithi022.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T10:52:18.317 INFO:teuthology.orchestra.run.smithi022.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T10:52:18.317 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:18.317 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:18.495 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:52:18.537 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T10:52:18.562 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.4 MB/s | 63 kB 00:00 2024-03-29T10:52:18.640 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-test 2024-03-29T10:52:18.704 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 430 kB/s | 163 kB 00:00 2024-03-29T10:52:19.227 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:29 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:19.384 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:19.385 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:52:19.385 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:19.385 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:52:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T10:52:19.385 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:52:19.385 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:19.386 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-29T10:52:19.387 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:19.388 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 48 M 2024-03-29T10:52:19.388 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 204 M 2024-03-29T10:52:19.388 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:52:19.397 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 224 kB/s | 187 kB 00:00 2024-03-29T10:52:19.633 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-29T10:52:19.735 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 3.0 MB/s | 300 kB 00:00 2024-03-29T10:52:19.767 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-29T10:52:19.959 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 977 kB/s | 187 kB 00:00 2024-03-29T10:52:20.006 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 300 kB/s | 163 kB 00:00 2024-03-29T10:52:20.905 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 18 MB/s | 47 MB 00:02 2024-03-29T10:52:20.908 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:20.908 INFO:teuthology.orchestra.run.smithi026.stdout:Total 17 MB/s | 48 MB 00:02 2024-03-29T10:52:20.924 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:52:20.946 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:52:20.946 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:52:21.340 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:52:21.340 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:52:21.924 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:52:22.006 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:52:22.050 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:52:22.082 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:52:22.123 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:52:22.149 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:52:23.592 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 11 MB/s | 47 MB 00:04 2024-03-29T10:52:23.595 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:23.596 INFO:teuthology.orchestra.run.smithi022.stdout:Total 11 MB/s | 48 MB 00:04 2024-03-29T10:52:23.610 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:52:23.633 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:52:23.633 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:52:24.032 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:52:24.032 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:52:24.633 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:52:24.710 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T10:52:24.761 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T10:52:24.809 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:52:24.851 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T10:52:24.879 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:52:26.869 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:52:26.943 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:52:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:52:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:52:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:52:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:52:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:52:29.279 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:52:29.579 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T10:52:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:52:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:29.621 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:52:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:29.621 INFO:teuthology.orchestra.run.smithi026.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:52:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:52:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:52:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:52:29.622 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:29.622 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:29.663 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:52:29.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph 2024-03-29T10:52:30.500 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:42 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:30.664 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:30.666 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-03-29T10:52:30.666 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:52:30.667 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:52:30.668 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout:========================================================================================= 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout:Install 36 Packages 2024-03-29T10:52:30.669 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:30.670 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 30 M 2024-03-29T10:52:30.670 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 107 M 2024-03-29T10:52:30.670 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:52:31.832 INFO:teuthology.orchestra.run.smithi026.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-29T10:52:31.973 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T10:52:31.973 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T10:52:31.973 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T10:52:31.973 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T10:52:31.973 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T10:52:31.973 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T10:52:32.099 INFO:teuthology.orchestra.run.smithi026.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 2.9 MB/s | 2.1 MB 00:00 2024-03-29T10:52:32.241 INFO:teuthology.orchestra.run.smithi026.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.7 MB/s | 1.5 MB 00:00 2024-03-29T10:52:32.367 INFO:teuthology.orchestra.run.smithi026.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.9 MB/s | 247 kB 00:00 2024-03-29T10:52:32.376 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.6-16.el9.x86_64 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:32.377 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:32.509 INFO:teuthology.orchestra.run.smithi026.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-29T10:52:32.550 INFO:teuthology.orchestra.run.smithi026.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:52:32.584 INFO:teuthology.orchestra.run.smithi026.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 950 kB/s | 32 kB 00:00 2024-03-29T10:52:32.617 INFO:teuthology.orchestra.run.smithi026.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-29T10:52:32.668 INFO:teuthology.orchestra.run.smithi026.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 773 kB/s | 39 kB 00:00 2024-03-29T10:52:32.705 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph 2024-03-29T10:52:32.718 INFO:teuthology.orchestra.run.smithi026.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T10:52:32.793 INFO:teuthology.orchestra.run.smithi026.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:52:32.961 INFO:teuthology.orchestra.run.smithi026.stdout:(12/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 19 MB/s | 17 MB 00:00 2024-03-29T10:52:32.986 INFO:teuthology.orchestra.run.smithi026.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 536 kB/s | 103 kB 00:00 2024-03-29T10:52:33.011 INFO:teuthology.orchestra.run.smithi026.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:52:33.036 INFO:teuthology.orchestra.run.smithi026.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.7 MB/s | 187 kB 00:00 2024-03-29T10:52:33.078 INFO:teuthology.orchestra.run.smithi026.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 538 kB/s | 22 kB 00:00 2024-03-29T10:52:33.178 INFO:teuthology.orchestra.run.smithi026.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 748 kB/s | 125 kB 00:00 2024-03-29T10:52:33.275 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:43 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:33.281 INFO:teuthology.orchestra.run.smithi026.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 747 kB/s | 151 kB 00:00 2024-03-29T10:52:33.331 INFO:teuthology.orchestra.run.smithi026.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 285 kB/s | 43 kB 00:00 2024-03-29T10:52:33.390 INFO:teuthology.orchestra.run.smithi026.stdout:(20/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 3.0 MB/s | 4.7 MB 00:01 2024-03-29T10:52:33.415 INFO:teuthology.orchestra.run.smithi026.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-29T10:52:33.435 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:33.437 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-29T10:52:33.437 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T10:52:33.438 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T10:52:33.439 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T10:52:33.440 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout:========================================================================================= 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout:Install 36 Packages 2024-03-29T10:52:33.441 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:33.442 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 30 M 2024-03-29T10:52:33.442 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 107 M 2024-03-29T10:52:33.442 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:52:33.474 INFO:teuthology.orchestra.run.smithi026.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-03-29T10:52:33.499 INFO:teuthology.orchestra.run.smithi026.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-29T10:52:33.524 INFO:teuthology.orchestra.run.smithi026.stdout:(24/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-29T10:52:33.549 INFO:teuthology.orchestra.run.smithi026.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T10:52:33.574 INFO:teuthology.orchestra.run.smithi026.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 257 kB/s | 19 kB 00:00 2024-03-29T10:52:33.600 INFO:teuthology.orchestra.run.smithi026.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-03-29T10:52:33.625 INFO:teuthology.orchestra.run.smithi026.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-03-29T10:52:33.675 INFO:teuthology.orchestra.run.smithi026.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-29T10:52:33.701 INFO:teuthology.orchestra.run.smithi026.stdout:(30/36): python3-more-itertools-8.12.0-2.el9.no 627 kB/s | 79 kB 00:00 2024-03-29T10:52:33.726 INFO:teuthology.orchestra.run.smithi026.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-29T10:52:33.751 INFO:teuthology.orchestra.run.smithi026.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-29T10:52:33.785 INFO:teuthology.orchestra.run.smithi026.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 612 kB/s | 36 kB 00:00 2024-03-29T10:52:33.835 INFO:teuthology.orchestra.run.smithi026.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-29T10:52:33.860 INFO:teuthology.orchestra.run.smithi026.stdout:(35/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-29T10:52:33.885 INFO:teuthology.orchestra.run.smithi026.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-29T10:52:33.890 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:33.890 INFO:teuthology.orchestra.run.smithi026.stdout:Total 9.2 MB/s | 30 MB 00:03 2024-03-29T10:52:33.915 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:52:33.961 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:52:33.961 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:52:34.138 INFO:teuthology.orchestra.run.smithi022.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 42 kB/s | 6.6 kB 00:00 2024-03-29T10:52:34.338 INFO:teuthology.orchestra.run.smithi022.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 4.2 MB/s | 1.5 MB 00:00 2024-03-29T10:52:34.406 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:52:34.406 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:52:34.656 INFO:teuthology.orchestra.run.smithi022.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 9.1 MB/s | 4.7 MB 00:00 2024-03-29T10:52:34.722 INFO:teuthology.orchestra.run.smithi022.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T10:52:34.973 INFO:teuthology.orchestra.run.smithi022.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 747 kB/s | 187 kB 00:00 2024-03-29T10:52:35.015 INFO:teuthology.orchestra.run.smithi022.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-29T10:52:35.040 INFO:teuthology.orchestra.run.smithi022.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.3 MB/s | 32 kB 00:00 2024-03-29T10:52:35.065 INFO:teuthology.orchestra.run.smithi022.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-29T10:52:35.107 INFO:teuthology.orchestra.run.smithi022.stdout:(9/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-03-29T10:52:35.124 INFO:teuthology.orchestra.run.smithi022.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 659 kB/s | 39 kB 00:00 2024-03-29T10:52:35.149 INFO:teuthology.orchestra.run.smithi022.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-29T10:52:35.199 INFO:teuthology.orchestra.run.smithi022.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-29T10:52:35.225 INFO:teuthology.orchestra.run.smithi022.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.3 MB/s | 103 kB 00:00 2024-03-29T10:52:35.250 INFO:teuthology.orchestra.run.smithi022.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T10:52:35.275 INFO:teuthology.orchestra.run.smithi022.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.6 MB/s | 187 kB 00:00 2024-03-29T10:52:35.434 INFO:teuthology.orchestra.run.smithi022.stdout:(16/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 15 MB/s | 17 MB 00:01 2024-03-29T10:52:35.597 INFO:teuthology.orchestra.run.smithi022.stdout:(17/36): lua-devel-5.4.4-4.el9.x86_64.rpm 69 kB/s | 22 kB 00:00 2024-03-29T10:52:35.670 INFO:teuthology.orchestra.run.smithi022.stdout:(18/36): python3-requests-2.25.1-8.el9.noarch.r 299 kB/s | 125 kB 00:00 2024-03-29T10:52:35.743 INFO:teuthology.orchestra.run.smithi022.stdout:(19/36): luarocks-3.9.2-1.el9.noarch.rpm 489 kB/s | 151 kB 00:00 2024-03-29T10:52:35.768 INFO:teuthology.orchestra.run.smithi022.stdout:(20/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 256 kB/s | 43 kB 00:00 2024-03-29T10:52:35.793 INFO:teuthology.orchestra.run.smithi022.stdout:(21/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-03-29T10:52:35.819 INFO:teuthology.orchestra.run.smithi022.stdout:(22/36): python3-cherrypy-18.6.1-2.el9.noarch.r 4.7 MB/s | 358 kB 00:00 2024-03-29T10:52:35.844 INFO:teuthology.orchestra.run.smithi022.stdout:(23/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 215 kB/s | 11 kB 00:00 2024-03-29T10:52:35.869 INFO:teuthology.orchestra.run.smithi022.stdout:(24/36): python3-jaraco-classes-3.2.1-5.el9.noa 236 kB/s | 18 kB 00:00 2024-03-29T10:52:35.895 INFO:teuthology.orchestra.run.smithi022.stdout:(25/36): python3-jaraco-collections-3.0.0-8.el9 307 kB/s | 23 kB 00:00 2024-03-29T10:52:35.919 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:52:35.920 INFO:teuthology.orchestra.run.smithi022.stdout:(26/36): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-03-29T10:52:35.937 INFO:teuthology.orchestra.run.smithi022.stdout:(27/36): python3-jaraco-text-3.2.0-6.el9.noarch 294 kB/s | 20 kB 00:00 2024-03-29T10:52:35.962 INFO:teuthology.orchestra.run.smithi022.stdout:(28/36): python3-logutils-0.3.5-21.el9.noarch.r 694 kB/s | 46 kB 00:00 2024-03-29T10:52:35.979 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:52:35.987 INFO:teuthology.orchestra.run.smithi022.stdout:(29/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T10:52:36.016 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:52:36.041 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:52:36.096 INFO:teuthology.orchestra.run.smithi022.stdout:(30/36): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-03-29T10:52:36.113 INFO:teuthology.orchestra.run.smithi022.stdout:(31/36): python3-portend-3.1.0-2.el9.noarch.rpm 109 kB/s | 16 kB 00:00 2024-03-29T10:52:36.114 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:52:36.146 INFO:teuthology.orchestra.run.smithi022.stdout:(32/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 568 kB/s | 90 kB 00:00 2024-03-29T10:52:36.156 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:52:36.163 INFO:teuthology.orchestra.run.smithi022.stdout:(33/36): python3-tempora-5.0.0-2.el9.noarch.rpm 718 kB/s | 36 kB 00:00 2024-03-29T10:52:36.188 INFO:teuthology.orchestra.run.smithi022.stdout:(34/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-03-29T10:52:36.189 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:52:36.230 INFO:teuthology.orchestra.run.smithi022.stdout:(35/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-29T10:52:36.247 INFO:teuthology.orchestra.run.smithi022.stdout:(36/36): python3-zc-lockfile-2.0-10.el9.noarch. 345 kB/s | 20 kB 00:00 2024-03-29T10:52:36.254 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:36.254 INFO:teuthology.orchestra.run.smithi022.stdout:Total 11 MB/s | 30 MB 00:02 2024-03-29T10:52:36.279 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:52:36.285 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:52:36.327 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:52:36.327 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:52:36.370 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:52:36.417 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:52:36.491 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:52:36.543 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:52:36.582 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:52:36.648 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:52:36.711 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:52:36.748 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:52:36.748 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:52:36.907 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:52:36.960 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:52:36.998 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:52:37.033 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:52:37.063 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:52:37.141 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:52:37.182 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:52:37.214 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:52:37.256 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:52:37.299 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:52:37.333 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:52:37.426 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:52:37.464 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:52:37.540 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:52:37.690 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:52:37.820 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:52:37.988 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:52:38.018 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:52:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:52:38.044 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:38.044 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:52:38.044 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:52:38.044 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:52:38.044 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:38.108 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:52:38.175 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T10:52:38.208 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T10:52:38.244 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T10:52:38.319 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T10:52:38.360 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T10:52:38.392 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T10:52:38.489 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T10:52:38.566 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T10:52:38.613 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T10:52:38.690 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T10:52:38.890 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T10:52:38.927 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T10:52:38.993 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T10:52:39.057 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T10:52:39.252 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T10:52:39.306 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T10:52:39.344 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T10:52:39.378 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T10:52:39.408 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T10:52:39.489 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T10:52:39.519 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:52:39.552 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T10:52:39.553 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:52:39.578 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:52:39.579 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:39.579 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:52:39.579 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:52:39.579 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:52:39.579 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:39.593 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T10:52:39.637 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T10:52:39.671 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T10:52:39.766 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T10:52:39.810 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T10:52:39.886 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T10:52:40.036 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:52:40.045 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T10:52:40.059 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:52:40.060 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:40.060 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:52:40.060 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:52:40.060 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:52:40.060 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:40.178 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:52:40.270 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:52:40.313 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:52:40.313 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:40.313 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:52:40.313 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:52:40.313 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:52:40.313 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:40.344 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T10:52:40.382 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:52:40.388 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:52:40.407 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T10:52:40.408 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:40.408 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T10:52:40.408 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:52:40.408 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T10:52:40.408 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:41.932 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:52:41.958 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T10:52:41.958 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:41.959 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T10:52:41.959 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:52:41.959 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T10:52:41.959 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:52:42.200 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:52:42.201 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:52:42.202 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:52:42.202 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:52:42.202 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:52:42.202 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:52:42.202 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:52:42.206 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:52:42.416 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:52:42.439 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T10:52:42.439 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:42.439 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T10:52:42.439 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:52:42.439 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T10:52:42.439 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:42.658 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:52:42.681 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T10:52:42.681 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:42.681 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T10:52:42.681 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:52:42.681 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T10:52:42.681 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:42.766 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:52:43.144 INFO:teuthology.orchestra.run.smithi026.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:52:43.145 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:43.146 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:43.482 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-base 2024-03-29T10:52:44.059 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:44.161 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:52:44.217 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:44.218 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:52:44.218 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:44.292 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install cephadm 2024-03-29T10:52:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T10:52:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T10:52:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T10:52:44.417 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T10:52:44.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T10:52:44.419 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T10:52:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T10:52:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T10:52:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T10:52:44.420 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T10:52:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T10:52:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T10:52:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T10:52:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T10:52:44.866 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:52:45.023 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:45.024 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-29T10:52:45.024 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:45.024 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 747 k 2024-03-29T10:52:45.024 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 754 k 2024-03-29T10:52:45.024 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:45.247 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T10:52:45.248 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T10:52:45.249 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T10:52:45.250 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:45.250 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:45.547 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base 2024-03-29T10:52:45.877 INFO:teuthology.orchestra.run.smithi026.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 877 kB/s | 747 kB 00:00 2024-03-29T10:52:45.878 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:45.878 INFO:teuthology.orchestra.run.smithi026.stdout:Total 875 kB/s | 747 kB 00:00 2024-03-29T10:52:45.879 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:52:45.885 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:52:45.885 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:52:45.891 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:52:45.891 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:52:46.084 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:52:46.141 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:46.242 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:52:46.297 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:46.298 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:52:46.298 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:46.361 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install cephadm 2024-03-29T10:52:46.590 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:46.677 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:46.933 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:56 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:47.021 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:47.090 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:47.091 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:47.091 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:52:47.091 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:47.091 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:52:47.091 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T10:52:47.091 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:47.091 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:52:47.092 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:47.092 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T10:52:47.092 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:47.092 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 747 k 2024-03-29T10:52:47.092 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 754 k 2024-03-29T10:52:47.092 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:52:47.325 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:47.325 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:47.325 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:52:47.326 INFO:teuthology.orchestra.run.smithi026.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:47.326 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:47.326 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:47.560 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:52:47.909 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 916 kB/s | 747 kB 00:00 2024-03-29T10:52:47.910 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:47.910 INFO:teuthology.orchestra.run.smithi022.stdout:Total 914 kB/s | 747 kB 00:00 2024-03-29T10:52:47.910 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:52:47.916 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:52:47.916 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:52:47.921 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:52:47.922 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:52:47.999 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:52:48.155 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:00 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:48.313 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:52:48.314 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:52:48.315 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-29T10:52:48.315 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:48.315 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 145 k 2024-03-29T10:52:48.315 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 436 k 2024-03-29T10:52:48.315 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:52:48.375 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:48.474 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 642 kB/s | 145 kB 00:00 2024-03-29T10:52:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:48.542 INFO:teuthology.orchestra.run.smithi026.stdout:Total 638 kB/s | 145 kB 00:00 2024-03-29T10:52:48.543 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:52:48.550 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:52:48.551 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:52:48.606 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:52:48.607 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:52:48.748 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:52:48.809 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:52:48.834 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:52:48.834 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:48.834 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:52:48.834 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:48.844 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T10:52:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:49.151 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:52:49.151 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:49.152 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:49.152 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:49.394 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T10:52:49.830 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:52:49.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:49.830 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:52:49.830 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:49.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:49.830 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:49.963 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:59 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:50.078 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr 2024-03-29T10:52:50.118 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:52:50.119 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:52:50.120 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T10:52:50.120 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:50.120 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 145 k 2024-03-29T10:52:50.120 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 436 k 2024-03-29T10:52:50.120 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:52:50.348 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 638 kB/s | 145 kB 00:00 2024-03-29T10:52:50.349 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:50.349 INFO:teuthology.orchestra.run.smithi022.stdout:Total 633 kB/s | 145 kB 00:00 2024-03-29T10:52:50.349 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:52:50.357 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:52:50.357 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:52:50.413 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:52:50.413 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:52:50.572 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:52:50.632 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:52:50.656 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:50.656 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:52:50.656 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:52:50.657 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T10:52:50.657 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:50.757 INFO:teuthology.orchestra.run.smithi026.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:52:50.813 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:50.814 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:52:50.814 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:50.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:52:51.453 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T10:52:51.584 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:51.584 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:52:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:52:51.585 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:51.585 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:51.610 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:51.611 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-03-29T10:52:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:52:51.611 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-03-29T10:52:51.611 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:52:51.611 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:52:51.611 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:52:51.612 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout:====================================================================================== 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-03-29T10:52:51.613 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:51.614 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 6.9 M 2024-03-29T10:52:51.614 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-03-29T10:52:51.614 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:52:51.860 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr 2024-03-29T10:52:52.328 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 127 kB/s | 31 kB 00:00 2024-03-29T10:52:52.353 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 56 kB/s | 15 kB 00:00 2024-03-29T10:52:52.436 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:02 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:52.495 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.0 MB/s | 238 kB 00:00 2024-03-29T10:52:52.529 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.0 MB/s | 1.0 MB 00:00 2024-03-29T10:52:52.535 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:52:52.589 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:52.591 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:52:52.591 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:52.612 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 3.4 MB/s | 1.8 MB 00:00 2024-03-29T10:52:52.660 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T10:52:52.830 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.5 MB/s | 863 kB 00:00 2024-03-29T10:52:52.854 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-29T10:52:52.980 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.2 MB/s | 548 kB 00:00 2024-03-29T10:52:53.080 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 307 kB/s | 31 kB 00:00 2024-03-29T10:52:53.105 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 576 kB/s | 144 kB 00:00 2024-03-29T10:52:53.180 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-29T10:52:53.227 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:03 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:53.383 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:53.384 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-29T10:52:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:52:53.384 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-29T10:52:53.384 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:52:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T10:52:53.384 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:52:53.384 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout:====================================================================================== 2024-03-29T10:52:53.385 INFO:teuthology.orchestra.run.smithi022.stdout:Install 12 Packages 2024-03-29T10:52:53.386 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:53.386 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.9 M 2024-03-29T10:52:53.386 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2024-03-29T10:52:53.386 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:52:53.739 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.2 MB/s | 2.0 MB 00:00 2024-03-29T10:52:53.742 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:53.742 INFO:teuthology.orchestra.run.smithi026.stdout:Total 3.2 MB/s | 6.9 MB 00:02 2024-03-29T10:52:53.761 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:52:53.780 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:52:53.780 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:52:53.819 INFO:teuthology.orchestra.run.smithi022.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 77 kB/s | 15 kB 00:00 2024-03-29T10:52:53.911 INFO:teuthology.orchestra.run.smithi022.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 108 kB/s | 31 kB 00:00 2024-03-29T10:52:53.958 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:52:53.958 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:52:53.969 INFO:teuthology.orchestra.run.smithi022.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-29T10:52:53.994 INFO:teuthology.orchestra.run.smithi022.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.8 MB/s | 238 kB 00:00 2024-03-29T10:52:54.036 INFO:teuthology.orchestra.run.smithi022.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.3 MB/s | 1.8 MB 00:00 2024-03-29T10:52:54.103 INFO:teuthology.orchestra.run.smithi022.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-29T10:52:54.237 INFO:teuthology.orchestra.run.smithi022.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.2 MB/s | 548 kB 00:00 2024-03-29T10:52:54.304 INFO:teuthology.orchestra.run.smithi022.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.1 MB/s | 144 kB 00:00 2024-03-29T10:52:54.371 INFO:teuthology.orchestra.run.smithi022.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-29T10:52:54.438 INFO:teuthology.orchestra.run.smithi022.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-29T10:52:54.493 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:52:54.573 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:52:54.613 INFO:teuthology.orchestra.run.smithi022.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 4.0 MB/s | 2.0 MB 00:00 2024-03-29T10:52:54.617 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:52:54.672 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:52:54.710 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:52:54.755 INFO:teuthology.orchestra.run.smithi022.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.1 MB/s | 863 kB 00:00 2024-03-29T10:52:54.758 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:54.759 INFO:teuthology.orchestra.run.smithi022.stdout:Total 5.0 MB/s | 6.9 MB 00:01 2024-03-29T10:52:54.778 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:52:54.797 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:52:54.797 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:52:54.813 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:52:54.949 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:52:54.983 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:52:54.984 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:52:55.050 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:52:55.098 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:52:55.126 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:52:55.167 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:52:55.422 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:52:55.450 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:52:55.501 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:52:55.589 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T10:52:55.633 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T10:52:55.680 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T10:52:55.718 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T10:52:55.824 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T10:52:55.956 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T10:52:56.050 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T10:52:56.098 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T10:52:56.126 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:52:56.157 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:52:56.233 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:52:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:52:56.234 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:52:56.412 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T10:52:56.441 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:52:56.575 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:52:56.575 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:56.575 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:52:56.575 INFO:teuthology.orchestra.run.smithi026.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:52:56.575 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:56.575 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:56.575 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:56.576 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:52:56.810 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:52:57.195 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T10:52:57.195 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T10:52:57.195 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T10:52:57.195 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T10:52:57.195 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T10:52:57.196 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T10:52:57.196 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T10:52:57.196 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T10:52:57.196 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T10:52:57.196 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T10:52:57.196 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T10:52:57.196 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T10:52:57.379 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:09 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:52:57.537 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:52:57.538 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-03-29T10:52:57.538 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:52:57.538 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-03-29T10:52:57.538 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:52:57.538 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:52:57.538 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:52:57.538 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:52:57.539 INFO:teuthology.orchestra.run.smithi026.stdout:============================================================================================ 2024-03-29T10:52:57.542 INFO:teuthology.orchestra.run.smithi026.stdout:Install 12 Packages 2024-03-29T10:52:57.542 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:52:57.542 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 37 M 2024-03-29T10:52:57.542 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 209 M 2024-03-29T10:52:57.542 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:52:57.558 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T10:52:57.558 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:57.558 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:57.559 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:52:57.835 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T10:52:57.837 INFO:teuthology.orchestra.run.smithi026.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 226 kB/s | 32 kB 00:00 2024-03-29T10:52:57.870 INFO:teuthology.orchestra.run.smithi026.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 536 kB/s | 17 kB 00:00 2024-03-29T10:52:57.912 INFO:teuthology.orchestra.run.smithi026.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 844 kB/s | 34 kB 00:00 2024-03-29T10:52:57.963 INFO:teuthology.orchestra.run.smithi026.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-29T10:52:58.146 INFO:teuthology.orchestra.run.smithi026.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T10:52:58.213 INFO:teuthology.orchestra.run.smithi026.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-29T10:52:58.238 INFO:teuthology.orchestra.run.smithi026.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T10:52:58.406 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:52:58.422 INFO:teuthology.orchestra.run.smithi026.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-29T10:52:58.566 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T10:52:58.567 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout:============================================================================================ 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout:Install 12 Packages 2024-03-29T10:52:58.568 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:52:58.569 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 37 M 2024-03-29T10:52:58.569 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 209 M 2024-03-29T10:52:58.569 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:52:58.572 INFO:teuthology.orchestra.run.smithi026.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-03-29T10:52:58.598 INFO:teuthology.orchestra.run.smithi026.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T10:52:59.090 INFO:teuthology.orchestra.run.smithi026.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 475 kB/s | 245 kB 00:00 2024-03-29T10:52:59.244 INFO:teuthology.orchestra.run.smithi022.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 231 kB/s | 32 kB 00:00 2024-03-29T10:52:59.256 INFO:teuthology.orchestra.run.smithi026.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.7 MB/s | 7.4 MB 00:01 2024-03-29T10:52:59.258 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:52:59.259 INFO:teuthology.orchestra.run.smithi026.stdout:Total 22 MB/s | 37 MB 00:01 2024-03-29T10:52:59.269 INFO:teuthology.orchestra.run.smithi022.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 722 kB/s | 17 kB 00:00 2024-03-29T10:52:59.295 INFO:teuthology.orchestra.run.smithi022.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-29T10:52:59.299 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:52:59.324 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:52:59.325 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:52:59.462 INFO:teuthology.orchestra.run.smithi022.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-29T10:52:59.554 INFO:teuthology.orchestra.run.smithi022.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.7 MB/s | 3.0 MB 00:00 2024-03-29T10:52:59.629 INFO:teuthology.orchestra.run.smithi022.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-29T10:52:59.654 INFO:teuthology.orchestra.run.smithi022.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-29T10:52:59.707 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:52:59.707 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:52:59.955 INFO:teuthology.orchestra.run.smithi022.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 48 MB/s | 16 MB 00:00 2024-03-29T10:53:00.005 INFO:teuthology.orchestra.run.smithi022.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-03-29T10:53:00.139 INFO:teuthology.orchestra.run.smithi022.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-29T10:53:00.231 INFO:teuthology.orchestra.run.smithi022.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.1 MB/s | 245 kB 00:00 2024-03-29T10:53:00.426 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:00.496 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:53:00.536 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:53:00.635 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:53:00.681 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:53:00.863 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:53:00.896 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:53:00.931 INFO:teuthology.orchestra.run.smithi022.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 4.1 MB/s | 7.4 MB 00:01 2024-03-29T10:53:00.933 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:00.934 INFO:teuthology.orchestra.run.smithi022.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-29T10:53:00.973 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:00.974 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:53:00.997 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:00.997 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:01.367 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:01.368 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:01.665 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:53:01.778 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:53:02.081 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:02.154 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T10:53:02.209 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T10:53:02.284 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:53:02.325 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T10:53:02.515 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:53:02.546 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:53:02.632 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T10:53:03.067 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:53:03.315 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T10:53:03.430 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T10:53:04.128 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:53:04.153 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:53:04.727 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T10:53:05.801 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T10:53:05.829 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:53:06.710 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:53:06.711 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:53:06.711 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:53:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:53:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:07.151 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:53:07.151 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:53:07.152 INFO:teuthology.orchestra.run.smithi026.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:53:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:53:07.153 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:07.153 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:07.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:53:08.104 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:20 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:08.262 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:53:08.263 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout:Install 13 Packages 2024-03-29T10:53:08.264 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:08.265 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 2.0 M 2024-03-29T10:53:08.265 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 26 M 2024-03-29T10:53:08.265 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:08.363 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T10:53:08.363 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T10:53:08.364 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T10:53:08.365 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T10:53:08.421 INFO:teuthology.orchestra.run.smithi026.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 175 kB/s | 18 kB 00:00 2024-03-29T10:53:08.472 INFO:teuthology.orchestra.run.smithi026.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 159 kB/s | 24 kB 00:00 2024-03-29T10:53:08.522 INFO:teuthology.orchestra.run.smithi026.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-29T10:53:08.547 INFO:teuthology.orchestra.run.smithi026.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:53:08.575 INFO:teuthology.orchestra.run.smithi026.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T10:53:08.598 INFO:teuthology.orchestra.run.smithi026.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 866 kB/s | 43 kB 00:00 2024-03-29T10:53:08.756 INFO:teuthology.orchestra.run.smithi026.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 119 kB/s | 52 kB 00:00 2024-03-29T10:53:08.777 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T10:53:08.778 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T10:53:08.779 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T10:53:08.779 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T10:53:08.779 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:08.779 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:08.832 INFO:teuthology.orchestra.run.smithi026.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-03-29T10:53:08.865 INFO:teuthology.orchestra.run.smithi026.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-29T10:53:08.957 INFO:teuthology.orchestra.run.smithi026.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.0 MB/s | 203 kB 00:00 2024-03-29T10:53:09.032 INFO:teuthology.orchestra.run.smithi026.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 354 kB/s | 59 kB 00:00 2024-03-29T10:53:09.074 INFO:teuthology.orchestra.run.smithi026.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 4.2 MB/s | 1.0 MB 00:00 2024-03-29T10:53:09.099 INFO:teuthology.orchestra.run.smithi026.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 632 kB/s | 90 kB 00:00 2024-03-29T10:53:09.103 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:09.103 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2024-03-29T10:53:09.103 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:09.118 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:09.118 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:09.127 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-rook 2024-03-29T10:53:09.216 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:09.216 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:09.699 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:19 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:09.741 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:09.857 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T10:53:09.858 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout:Install 13 Packages 2024-03-29T10:53:09.859 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:09.860 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 2.0 M 2024-03-29T10:53:09.860 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2024-03-29T10:53:09.860 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:09.920 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:53:09.978 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:53:10.042 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:53:10.071 INFO:teuthology.orchestra.run.smithi022.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 151 kB/s | 18 kB 00:00 2024-03-29T10:53:10.077 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:53:10.097 INFO:teuthology.orchestra.run.smithi022.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 168 kB/s | 24 kB 00:00 2024-03-29T10:53:10.119 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:53:10.147 INFO:teuthology.orchestra.run.smithi022.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T10:53:10.172 INFO:teuthology.orchestra.run.smithi022.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-29T10:53:10.198 INFO:teuthology.orchestra.run.smithi022.stdout:(5/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 212 kB/s | 52 kB 00:00 2024-03-29T10:53:10.223 INFO:teuthology.orchestra.run.smithi022.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.7 MB/s | 211 kB 00:00 2024-03-29T10:53:10.234 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:53:10.248 INFO:teuthology.orchestra.run.smithi022.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 570 kB/s | 43 kB 00:00 2024-03-29T10:53:10.350 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:53:10.405 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:53:10.473 INFO:teuthology.orchestra.run.smithi022.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-29T10:53:10.532 INFO:teuthology.orchestra.run.smithi022.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 104 kB/s | 32 kB 00:00 2024-03-29T10:53:10.624 INFO:teuthology.orchestra.run.smithi022.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 539 kB/s | 203 kB 00:00 2024-03-29T10:53:10.657 INFO:teuthology.orchestra.run.smithi022.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 472 kB/s | 59 kB 00:00 2024-03-29T10:53:10.707 INFO:teuthology.orchestra.run.smithi022.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.1 MB/s | 90 kB 00:00 2024-03-29T10:53:10.783 INFO:teuthology.orchestra.run.smithi022.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.3 MB/s | 1.0 MB 00:00 2024-03-29T10:53:10.787 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:10.787 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-29T10:53:10.788 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:10.802 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:10.803 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:10.899 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:10.900 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:11.170 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:53:11.217 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:53:11.242 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:53:11.291 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:53:11.320 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:53:11.428 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:11.612 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T10:53:11.677 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T10:53:11.741 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T10:53:11.777 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T10:53:11.819 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T10:53:11.916 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T10:53:12.026 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T10:53:12.079 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:53:12.105 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:53:12.106 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:53:12.466 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:12.467 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:12.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:53:12.856 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T10:53:12.900 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T10:53:12.934 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T10:53:12.983 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T10:53:13.012 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:53:13.327 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:13.486 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:53:13.487 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:53:13.488 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:53:13.488 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:13.488 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:13.488 INFO:teuthology.orchestra.run.smithi026.stdout:======================================================================================== 2024-03-29T10:53:13.488 INFO:teuthology.orchestra.run.smithi026.stdout:Install 6 Packages 2024-03-29T10:53:13.488 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:13.489 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 6.8 M 2024-03-29T10:53:13.489 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 33 M 2024-03-29T10:53:13.489 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T10:53:13.786 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T10:53:13.787 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T10:53:13.787 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T10:53:13.787 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T10:53:13.926 INFO:teuthology.orchestra.run.smithi026.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.9 MB/s | 248 kB 00:00 2024-03-29T10:53:14.076 INFO:teuthology.orchestra.run.smithi026.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-29T10:53:14.164 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T10:53:14.164 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:14.164 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:14.164 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T10:53:14.165 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:14.166 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:14.185 INFO:teuthology.orchestra.run.smithi026.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 354 kB/s | 138 kB 00:00 2024-03-29T10:53:14.210 INFO:teuthology.orchestra.run.smithi026.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 433 kB/s | 58 kB 00:00 2024-03-29T10:53:14.235 INFO:teuthology.orchestra.run.smithi026.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.7 MB/s | 548 kB 00:00 2024-03-29T10:53:14.260 INFO:teuthology.orchestra.run.smithi026.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 680 kB/s | 51 kB 00:00 2024-03-29T10:53:14.263 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:14.263 INFO:teuthology.orchestra.run.smithi026.stdout:Total 8.8 MB/s | 6.8 MB 00:00 2024-03-29T10:53:14.276 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:14.286 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:14.287 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:14.378 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:14.379 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:14.432 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T10:53:14.681 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:14.856 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:53:14.913 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:53:15.013 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:25 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:15.172 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T10:53:15.173 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T10:53:15.174 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:15.174 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:15.174 INFO:teuthology.orchestra.run.smithi022.stdout:======================================================================================== 2024-03-29T10:53:15.174 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2024-03-29T10:53:15.174 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:15.175 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.8 M 2024-03-29T10:53:15.175 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 33 M 2024-03-29T10:53:15.175 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:15.462 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:53:15.550 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:53:15.620 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:53:15.649 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:53:15.777 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-29T10:53:15.878 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-29T10:53:15.903 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 473 kB/s | 138 kB 00:00 2024-03-29T10:53:16.028 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 384 kB/s | 58 kB 00:00 2024-03-29T10:53:16.053 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 339 kB/s | 51 kB 00:00 2024-03-29T10:53:16.137 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-03-29T10:53:16.138 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:16.139 INFO:teuthology.orchestra.run.smithi022.stdout:Total 7.0 MB/s | 6.8 MB 00:00 2024-03-29T10:53:16.152 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:16.163 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:16.163 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:16.255 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:16.255 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:16.458 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:53:16.459 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:53:16.459 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:53:16.459 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:53:16.459 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:53:16.459 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:53:16.554 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:16.721 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T10:53:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:53:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:16.763 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:16.763 INFO:teuthology.orchestra.run.smithi026.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:53:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:53:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:53:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:53:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:53:16.764 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:16.764 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:16.773 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T10:53:16.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-fuse 2024-03-29T10:53:17.340 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T10:53:17.426 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T10:53:17.488 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T10:53:17.542 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:53:17.548 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:17.703 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-03-29T10:53:17.704 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:17.705 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 964 k 2024-03-29T10:53:17.705 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 2.8 M 2024-03-29T10:53:17.705 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:17.837 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 599 kB/s | 79 kB 00:00 2024-03-29T10:53:18.277 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T10:53:18.278 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T10:53:18.278 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T10:53:18.278 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T10:53:18.278 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T10:53:18.278 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T10:53:18.587 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T10:53:18.587 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:18.587 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:18.587 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:18.587 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T10:53:18.587 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T10:53:18.588 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T10:53:18.588 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T10:53:18.588 INFO:teuthology.orchestra.run.smithi022.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T10:53:18.588 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:18.588 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:18.834 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-fuse 2024-03-29T10:53:18.862 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 765 kB/s | 885 kB 00:01 2024-03-29T10:53:18.863 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:18.864 INFO:teuthology.orchestra.run.smithi026.stdout:Total 832 kB/s | 964 kB 00:01 2024-03-29T10:53:18.864 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:18.889 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:18.889 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:18.970 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:18.970 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:19.156 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:19.300 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:53:19.372 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:53:19.404 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:29 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:19.561 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:19.561 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:19.561 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:53:19.561 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:19.561 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-29T10:53:19.562 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:19.563 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 964 k 2024-03-29T10:53:19.563 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 2.8 M 2024-03-29T10:53:19.563 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:19.689 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 626 kB/s | 79 kB 00:00 2024-03-29T10:53:20.042 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:53:20.042 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:53:20.355 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:53:20.356 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:20.356 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:20.356 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:53:20.356 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:20.356 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:20.658 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install ceph-volume 2024-03-29T10:53:20.805 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 712 kB/s | 885 kB 00:01 2024-03-29T10:53:20.807 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:20.807 INFO:teuthology.orchestra.run.smithi022.stdout:Total 774 kB/s | 964 kB 00:01 2024-03-29T10:53:20.807 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:20.833 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:20.833 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:20.913 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:20.913 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:21.105 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:21.244 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T10:53:21.246 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:21.326 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:53:21.405 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:21.405 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:21.405 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repository Size 2024-03-29T10:53:21.405 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout:Installing dependencies: 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout:Install 2 Packages 2024-03-29T10:53:21.406 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:21.407 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 332 k 2024-03-29T10:53:21.407 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 1.5 M 2024-03-29T10:53:21.407 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:21.517 INFO:teuthology.orchestra.run.smithi026.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 625 kB/s | 69 kB 00:00 2024-03-29T10:53:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T10:53:21.989 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T10:53:22.279 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T10:53:22.279 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:22.279 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:22.280 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T10:53:22.280 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:22.280 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:22.317 INFO:teuthology.orchestra.run.smithi026.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 290 kB/s | 264 kB 00:00 2024-03-29T10:53:22.318 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:22.319 INFO:teuthology.orchestra.run.smithi026.stdout:Total 364 kB/s | 332 kB 00:00 2024-03-29T10:53:22.319 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:22.329 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:22.330 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:22.377 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:22.377 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:22.534 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:22.567 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-volume 2024-03-29T10:53:22.724 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:53:22.761 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:53:22.777 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:53:22.777 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:53:22.778 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:53:22.778 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:23.149 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:33 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:23.308 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:23.308 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:23.308 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2024-03-29T10:53:23.308 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:23.308 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:23.308 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T10:53:23.308 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2024-03-29T10:53:23.309 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T10:53:23.309 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:23.309 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:23.309 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:23.309 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2024-03-29T10:53:23.309 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:23.309 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 332 k 2024-03-29T10:53:23.310 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.5 M 2024-03-29T10:53:23.310 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:23.418 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 635 kB/s | 69 kB 00:00 2024-03-29T10:53:23.494 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:53:23.830 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:53:23.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:23.830 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:23.830 INFO:teuthology.orchestra.run.smithi026.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:23.830 INFO:teuthology.orchestra.run.smithi026.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:53:23.830 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:23.830 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:24.077 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 344 kB/s | 264 kB 00:00 2024-03-29T10:53:24.078 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:24.078 INFO:teuthology.orchestra.run.smithi022.stdout:Total 432 kB/s | 332 kB 00:00 2024-03-29T10:53:24.078 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:24.089 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:24.089 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:24.137 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:24.138 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:24.165 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados-devel 2024-03-29T10:53:24.290 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:24.485 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T10:53:24.513 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:53:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T10:53:24.529 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:53:24.529 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T10:53:24.529 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:24.755 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:24.911 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:24.912 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:24.912 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-29T10:53:24.912 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:24.913 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-29T10:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 127 k 2024-03-29T10:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 456 k 2024-03-29T10:53:24.914 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:25.348 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T10:53:25.489 INFO:teuthology.orchestra.run.smithi026.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 220 kB/s | 127 kB 00:00 2024-03-29T10:53:25.490 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:25.490 INFO:teuthology.orchestra.run.smithi026.stdout:Total 219 kB/s | 127 kB 00:00 2024-03-29T10:53:25.490 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:25.497 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:25.497 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:25.559 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:25.559 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:25.653 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T10:53:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:25.654 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T10:53:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T10:53:25.654 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:25.654 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:25.710 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:25.807 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:25.985 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados-devel 2024-03-29T10:53:26.241 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:26.514 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:26.515 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:26.515 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:26.515 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:26.573 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:26.730 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:26.731 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:26.731 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T10:53:26.731 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:26.731 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:26.731 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 127 k 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 456 k 2024-03-29T10:53:26.732 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:26.747 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs2 2024-03-29T10:53:27.258 INFO:teuthology.orchestra.run.smithi022.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 241 kB/s | 127 kB 00:00 2024-03-29T10:53:27.258 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:27.259 INFO:teuthology.orchestra.run.smithi022.stdout:Total 240 kB/s | 127 kB 00:00 2024-03-29T10:53:27.259 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:27.265 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:27.265 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:27.325 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:27.325 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:27.333 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:27.437 INFO:teuthology.orchestra.run.smithi026.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:27.475 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:27.493 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:27.494 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:53:27.494 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:27.565 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install libcephfs-devel 2024-03-29T10:53:27.576 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:28.009 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:28.145 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:40 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:28.303 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:28.303 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:28.303 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:28.304 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:28.304 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:28.304 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:28.304 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 31 k 2024-03-29T10:53:28.305 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 140 k 2024-03-29T10:53:28.306 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:28.553 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs2 2024-03-29T10:53:28.719 INFO:teuthology.orchestra.run.smithi026.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 75 kB/s | 31 kB 00:00 2024-03-29T10:53:28.720 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:28.720 INFO:teuthology.orchestra.run.smithi026.stdout:Total 75 kB/s | 31 kB 00:00 2024-03-29T10:53:28.720 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:28.724 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:28.724 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:28.749 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:28.749 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:28.884 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:28.987 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:29.132 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:29.234 INFO:teuthology.orchestra.run.smithi022.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:29.290 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:29.291 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:53:29.292 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:29.354 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:29.367 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs-devel 2024-03-29T10:53:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:29.619 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:29.619 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:29.619 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:29.784 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librados2 2024-03-29T10:53:29.945 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:30.104 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:30.105 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:30.105 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T10:53:30.105 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:30.105 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 31 k 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 140 k 2024-03-29T10:53:30.106 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:30.354 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:30.454 INFO:teuthology.orchestra.run.smithi026.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:30.509 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:30.510 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:53:30.510 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:30.520 INFO:teuthology.orchestra.run.smithi022.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 75 kB/s | 31 kB 00:00 2024-03-29T10:53:30.521 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:30.521 INFO:teuthology.orchestra.run.smithi022.stdout:Total 75 kB/s | 31 kB 00:00 2024-03-29T10:53:30.521 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:30.525 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:30.525 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:30.555 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install librbd1 2024-03-29T10:53:30.556 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:30.556 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:30.682 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:30.806 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:31.130 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:31.175 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:31.230 INFO:teuthology.orchestra.run.smithi026.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:31.284 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:31.285 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:53:31.286 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:31.331 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rados 2024-03-29T10:53:31.452 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:31.452 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:31.453 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:31.453 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:31.453 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:31.453 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:31.615 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados2 2024-03-29T10:53:31.909 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:32.010 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:32.065 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:32.066 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:53:32.066 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:32.115 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rgw 2024-03-29T10:53:32.184 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:32.284 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:32.339 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:32.340 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:53:32.340 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:32.388 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librbd1 2024-03-29T10:53:32.695 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:32.796 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:32.852 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:32.853 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:53:32.853 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:32.907 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-cephfs 2024-03-29T10:53:32.961 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:33.063 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:33.117 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:33.118 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:53:33.119 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:33.175 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rados 2024-03-29T10:53:33.484 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:33.585 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:33.640 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:33.641 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:53:33.641 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:33.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install python3-rbd 2024-03-29T10:53:33.750 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:43 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:33.853 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:33.908 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:33.909 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:53:33.909 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:33.957 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rgw 2024-03-29T10:53:34.269 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:34.374 INFO:teuthology.orchestra.run.smithi026.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:34.430 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:34.431 INFO:teuthology.orchestra.run.smithi026.stdout:Nothing to do. 2024-03-29T10:53:34.431 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:34.486 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-fuse 2024-03-29T10:53:34.532 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:44 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:34.639 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:34.696 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:34.697 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:53:34.697 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:34.750 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-cephfs 2024-03-29T10:53:35.069 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:35.225 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:35.226 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-29T10:53:35.226 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:35.226 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:53:35.226 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:35.226 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:35.227 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:35.227 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-29T10:53:35.227 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:35.227 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 85 k 2024-03-29T10:53:35.227 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 231 k 2024-03-29T10:53:35.227 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:35.331 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:45 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:35.433 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:35.489 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:35.490 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:53:35.490 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:35.539 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 273 kB/s | 85 kB 00:00 2024-03-29T10:53:35.540 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:35.540 INFO:teuthology.orchestra.run.smithi026.stdout:Total 272 kB/s | 85 kB 00:00 2024-03-29T10:53:35.541 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:35.541 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rbd 2024-03-29T10:53:35.548 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:35.548 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:35.606 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:35.606 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:35.750 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:35.849 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:36.118 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:36.220 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T10:53:36.252 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:36.274 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:36.276 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2024-03-29T10:53:36.276 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:36.330 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-fuse 2024-03-29T10:53:36.556 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:36.556 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:36.556 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:36.556 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:36.556 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:36.556 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:36.793 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-mirror 2024-03-29T10:53:36.904 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:46 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:37.059 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 85 k 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 231 k 2024-03-29T10:53:37.060 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:37.374 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:37.399 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 252 kB/s | 85 kB 00:00 2024-03-29T10:53:37.399 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:37.399 INFO:teuthology.orchestra.run.smithi022.stdout:Total 251 kB/s | 85 kB 00:00 2024-03-29T10:53:37.400 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:37.407 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:37.407 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:37.465 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:37.465 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:37.531 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout: Package Arch Version Repo Size 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-29T10:53:37.532 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:37.533 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 3.1 M 2024-03-29T10:53:37.533 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 13 M 2024-03-29T10:53:37.533 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:37.604 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:37.693 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:38.105 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:38.418 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:38.418 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:38.418 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:38.659 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-mirror 2024-03-29T10:53:38.734 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.6 MB/s | 3.1 MB 00:01 2024-03-29T10:53:38.734 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:38.735 INFO:teuthology.orchestra.run.smithi026.stdout:Total 2.6 MB/s | 3.1 MB 00:01 2024-03-29T10:53:38.735 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:38.747 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:38.747 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:38.827 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:38.827 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:39.241 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:49 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:39.282 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:39.343 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:39.367 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:39.367 INFO:teuthology.orchestra.run.smithi026.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:53:39.367 INFO:teuthology.orchestra.run.smithi026.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:53:39.367 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:53:39.367 INFO:teuthology.orchestra.run.smithi026.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:53:39.368 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:39.398 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:39.398 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 3.1 M 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 13 M 2024-03-29T10:53:39.399 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:40.469 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:40.469 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:40.469 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:40.469 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:40.469 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:40.469 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:40.627 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.5 MB/s | 3.1 MB 00:01 2024-03-29T10:53:40.628 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:40.628 INFO:teuthology.orchestra.run.smithi022.stdout:Total 2.5 MB/s | 3.1 MB 00:01 2024-03-29T10:53:40.628 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:40.640 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:40.640 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:40.720 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:40.720 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:40.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo yum -y install rbd-nbd 2024-03-29T10:53:41.193 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:41.245 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:41.270 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:41.271 INFO:teuthology.orchestra.run.smithi022.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T10:53:41.271 INFO:teuthology.orchestra.run.smithi022.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T10:53:41.271 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:53:41.271 INFO:teuthology.orchestra.run.smithi022.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T10:53:41.271 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:41.309 INFO:teuthology.orchestra.run.smithi026.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:51:48 AM UTC. 2024-03-29T10:53:41.465 INFO:teuthology.orchestra.run.smithi026.stdout:Dependencies resolved. 2024-03-29T10:53:41.466 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:41.466 INFO:teuthology.orchestra.run.smithi026.stdout: Package Architecture Version Repository Size 2024-03-29T10:53:41.466 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:41.466 INFO:teuthology.orchestra.run.smithi026.stdout:Installing: 2024-03-29T10:53:41.466 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:53:41.466 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:41.466 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction Summary 2024-03-29T10:53:41.467 INFO:teuthology.orchestra.run.smithi026.stdout:================================================================================ 2024-03-29T10:53:41.467 INFO:teuthology.orchestra.run.smithi026.stdout:Install 1 Package 2024-03-29T10:53:41.467 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:41.467 INFO:teuthology.orchestra.run.smithi026.stdout:Total download size: 171 k 2024-03-29T10:53:41.467 INFO:teuthology.orchestra.run.smithi026.stdout:Installed size: 494 k 2024-03-29T10:53:41.467 INFO:teuthology.orchestra.run.smithi026.stdout:Downloading Packages: 2024-03-29T10:53:41.670 INFO:teuthology.orchestra.run.smithi026.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 847 kB/s | 171 kB 00:00 2024-03-29T10:53:41.670 INFO:teuthology.orchestra.run.smithi026.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:41.670 INFO:teuthology.orchestra.run.smithi026.stdout:Total 841 kB/s | 171 kB 00:00 2024-03-29T10:53:41.671 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction check 2024-03-29T10:53:41.679 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction check succeeded. 2024-03-29T10:53:41.679 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction test 2024-03-29T10:53:41.737 INFO:teuthology.orchestra.run.smithi026.stdout:Transaction test succeeded. 2024-03-29T10:53:41.738 INFO:teuthology.orchestra.run.smithi026.stdout:Running transaction 2024-03-29T10:53:41.890 INFO:teuthology.orchestra.run.smithi026.stdout: Preparing : 1/1 2024-03-29T10:53:42.005 INFO:teuthology.orchestra.run.smithi026.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:42.413 INFO:teuthology.orchestra.run.smithi026.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:42.433 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:42.433 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:42.433 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:42.433 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:42.434 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:42.434 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:42.667 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-nbd 2024-03-29T10:53:42.712 INFO:teuthology.orchestra.run.smithi026.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:42.712 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:42.712 INFO:teuthology.orchestra.run.smithi026.stdout:Installed: 2024-03-29T10:53:42.713 INFO:teuthology.orchestra.run.smithi026.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:42.713 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:42.713 INFO:teuthology.orchestra.run.smithi026.stdout:Complete! 2024-03-29T10:53:42.940 DEBUG:teuthology.parallel:result is None 2024-03-29T10:53:43.246 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:53 ago on Fri 29 Mar 2024 10:51:50 AM UTC. 2024-03-29T10:53:43.400 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2024-03-29T10:53:43.401 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2024-03-29T10:53:43.402 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2024-03-29T10:53:43.402 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:43.402 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 171 k 2024-03-29T10:53:43.402 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 494 k 2024-03-29T10:53:43.402 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2024-03-29T10:53:43.631 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 752 kB/s | 171 kB 00:00 2024-03-29T10:53:43.631 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2024-03-29T10:53:43.631 INFO:teuthology.orchestra.run.smithi022.stdout:Total 747 kB/s | 171 kB 00:00 2024-03-29T10:53:43.632 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2024-03-29T10:53:43.640 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2024-03-29T10:53:43.640 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2024-03-29T10:53:43.699 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2024-03-29T10:53:43.699 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2024-03-29T10:53:43.855 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2024-03-29T10:53:43.962 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:44.370 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T10:53:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:44.677 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2024-03-29T10:53:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T10:53:44.677 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:44.677 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2024-03-29T10:53:44.903 DEBUG:teuthology.parallel:result is None 2024-03-29T10:53:44.904 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:53:45.120 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:53:45.140 INFO:teuthology.orchestra.run.smithi022.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:53:45.141 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:53:45.141 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:53:45.143 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T10:53:45.355 DEBUG:teuthology.orchestra.run.smithi026:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T10:53:45.379 INFO:teuthology.orchestra.run.smithi026.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T10:53:45.380 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T10:53:45.380 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T10:53:45.382 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T10:53:45.382 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:45.382 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:53:45.411 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:53:45.411 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T10:53:45.448 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T10:53:45.449 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:45.449 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:53:45.478 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:53:45.546 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:53:45.546 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T10:53:45.573 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T10:53:45.641 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T10:53:45.641 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:45.642 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:53:45.669 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:53:45.735 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:53:45.735 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T10:53:45.762 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T10:53:45.829 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T10:53:45.829 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:45.829 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:53:45.856 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:53:45.922 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:53:45.922 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T10:53:45.949 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T10:53:46.016 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T10:53:46.109 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T10:53:46.109 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /var/log/ceph 2024-03-29T10:53:46.111 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 777 /var/log/ceph 2024-03-29T10:53:46.137 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T10:53:46.138 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:53:46.178 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T10:53:46.204 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T10:53:46.204 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:53:46.245 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T10:53:46.282 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T10:53:46.282 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T10:53:46.282 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622663', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi022.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyACxNVF/D20cEsz4I+IiCBGhDOKyJdc1ozs7gI/OY9v8Yi188AgmNl4XqE171C391oJoQ0WCjrZbTwX7R5NnY=', 'smithi026.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwmYzMU7GD+l5piL0UNy8nu1PbS3hfMr10PIsafCqxd5VMKzXTWrdcyK3b1TFXRWHs5lh+mJVBBao2kK5ilz54='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T10:53:46.283 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:53:46.313 DEBUG:teuthology.orchestra.run.smithi026:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T10:53:46.328 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:53:46.368 DEBUG:teuthology.orchestra.run.smithi026:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T10:53:46.395 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:46.395 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:53:46.447 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:53:46.447 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2024-03-29T10:53:46.501 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:53:46.501 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:46.501 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-29T10:53:46.501 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:46.501 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:46.501 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 10:53:41.544921143 +0000 2024-03-29T10:53:46.502 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 10:50:54.125127789 +0000 2024-03-29T10:53:46.502 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 10:50:54.125127789 +0000 2024-03-29T10:53:46.502 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 10:50:54.125127789 +0000 2024-03-29T10:53:46.502 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:53:46.567 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T10:53:46.567 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T10:53:46.567 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 8.7416e-05 s, 5.9 MB/s 2024-03-29T10:53:46.568 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:53:46.623 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2024-03-29T10:53:46.677 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:53:46.677 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:46.677 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-29T10:53:46.678 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:46.678 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:46.678 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 10:53:41.544921143 +0000 2024-03-29T10:53:46.678 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 10:50:54.454123173 +0000 2024-03-29T10:53:46.678 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 10:50:54.454123173 +0000 2024-03-29T10:53:46.678 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 10:50:54.454123173 +0000 2024-03-29T10:53:46.678 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:53:46.743 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T10:53:46.743 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T10:53:46.743 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 9.4877e-05 s, 5.4 MB/s 2024-03-29T10:53:46.744 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:53:46.799 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 10:53:41.545921130 +0000 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 10:50:54.788118487 +0000 2024-03-29T10:53:46.853 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 10:50:54.788118487 +0000 2024-03-29T10:53:46.854 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 10:50:54.788118487 +0000 2024-03-29T10:53:46.854 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:53:46.919 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T10:53:46.919 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T10:53:46.919 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 7.6496e-05 s, 6.7 MB/s 2024-03-29T10:53:46.921 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:53:46.977 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2024-03-29 10:53:41.545921130 +0000 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2024-03-29 10:50:55.291111430 +0000 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2024-03-29 10:50:55.291111430 +0000 2024-03-29T10:53:47.034 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: 2024-03-29 10:50:55.291111430 +0000 2024-03-29T10:53:47.035 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:53:47.099 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2024-03-29T10:53:47.099 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2024-03-29T10:53:47.099 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 8.1864e-05 s, 6.3 MB/s 2024-03-29T10:53:47.100 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:53:47.155 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:53:47.155 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:53:47.155 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T10:53:47.173 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T10:53:47.174 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_1 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-29 10:53:39.629854350 +0000 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-29 10:50:54.126233103 +0000 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-29 10:50:54.126233103 +0000 2024-03-29T10:53:47.235 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-29 10:50:54.126233103 +0000 2024-03-29T10:53:47.236 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T10:53:47.304 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-29T10:53:47.304 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-29T10:53:47.304 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000112791 s, 4.5 MB/s 2024-03-29T10:53:47.305 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T10:53:47.364 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_2 2024-03-29T10:53:47.421 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T10:53:47.421 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:47.421 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T10:53:47.422 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:47.422 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:47.422 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-29 10:53:39.629854350 +0000 2024-03-29T10:53:47.422 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-29 10:50:54.461228400 +0000 2024-03-29T10:53:47.422 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-29 10:50:54.461228400 +0000 2024-03-29T10:53:47.422 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-29 10:50:54.461228400 +0000 2024-03-29T10:53:47.422 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T10:53:47.489 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-29T10:53:47.490 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-29T10:53:47.490 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000106206 s, 4.8 MB/s 2024-03-29T10:53:47.491 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T10:53:47.549 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_3 2024-03-29T10:53:47.604 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T10:53:47.605 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:47.605 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-29T10:53:47.605 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:47.605 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:47.605 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-29 10:53:39.630854336 +0000 2024-03-29T10:53:47.605 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-29 10:50:54.792223753 +0000 2024-03-29T10:53:47.605 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-29 10:50:54.792223753 +0000 2024-03-29T10:53:47.606 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-29 10:50:54.792223753 +0000 2024-03-29T10:53:47.606 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T10:53:47.672 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-29T10:53:47.672 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-29T10:53:47.672 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 9.9936e-05 s, 5.1 MB/s 2024-03-29T10:53:47.673 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T10:53:47.730 DEBUG:teuthology.orchestra.run.smithi026:> stat /dev/vg_nvme/lv_4 2024-03-29T10:53:47.786 INFO:teuthology.orchestra.run.smithi026.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T10:53:47.786 INFO:teuthology.orchestra.run.smithi026.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T10:53:47.786 INFO:teuthology.orchestra.run.smithi026.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-29T10:53:47.786 INFO:teuthology.orchestra.run.smithi026.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T10:53:47.787 INFO:teuthology.orchestra.run.smithi026.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T10:53:47.787 INFO:teuthology.orchestra.run.smithi026.stdout:Access: 2024-03-29 10:53:39.630854336 +0000 2024-03-29T10:53:47.787 INFO:teuthology.orchestra.run.smithi026.stdout:Modify: 2024-03-29 10:50:55.110219289 +0000 2024-03-29T10:53:47.787 INFO:teuthology.orchestra.run.smithi026.stdout:Change: 2024-03-29 10:50:55.110219289 +0000 2024-03-29T10:53:47.787 INFO:teuthology.orchestra.run.smithi026.stdout: Birth: 2024-03-29 10:50:55.110219289 +0000 2024-03-29T10:53:47.787 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T10:53:47.853 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records in 2024-03-29T10:53:47.853 INFO:teuthology.orchestra.run.smithi026.stderr:1+0 records out 2024-03-29T10:53:47.853 INFO:teuthology.orchestra.run.smithi026.stderr:512 bytes copied, 0.000108478 s, 4.7 MB/s 2024-03-29T10:53:47.854 DEBUG:teuthology.orchestra.run.smithi026:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T10:53:47.910 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:53:47.911 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:53:47.911 INFO:tasks.ceph:Generating config... 2024-03-29T10:53:47.912 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T10:53:47.912 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T10:53:47.912 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T10:53:47.912 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T10:53:47.912 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T10:53:47.912 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T10:53:47.912 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T10:53:47.912 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T10:53:47.912 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T10:53:47.912 INFO:tasks.ceph:[global] ms type = async 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T10:53:47.913 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T10:53:47.913 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T10:53:47.913 INFO:tasks.ceph:[osd] bluestore compression algorithm = zstd 2024-03-29T10:53:47.913 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T10:53:47.914 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T10:53:47.915 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T10:53:47.915 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T10:53:47.915 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T10:53:47.969 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T10:53:47.971 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T10:53:48.017 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:53:48.044 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.22:6789]'), ('mon.c', '[v1:172.21.15.22:6790]'), ('mon.b', '[v1:172.21.15.26:6789]')] 2024-03-29T10:53:48.045 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.26:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'zstd', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T10:53:48.046 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:48.046 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:53:48.100 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.22:6789]' --addv c '[v1:172.21.15.22:6790]' --addv b '[v1:172.21.15.26:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:48.184 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:48.184 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: generated fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:53:48.184 INFO:teuthology.orchestra.run.smithi022.stdout:setting min_mon_release = quincy 2024-03-29T10:53:48.184 INFO:teuthology.orchestra.run.smithi022.stdout:epoch 0 2024-03-29T10:53:48.184 INFO:teuthology.orchestra.run.smithi022.stdout:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:min_mon_release 17 (quincy) 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:election_strategy: 1 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:53:48.185 INFO:teuthology.orchestra.run.smithi022.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T10:53:48.186 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T10:53:48.200 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9704968d-8e2e-420e-abdf-0499c0fc035c... 2024-03-29T10:53:48.201 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:53:48.243 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T10:53:48.291 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2024-03-29T10:53:48.291 INFO:teuthology.orchestra.run.smithi022.stdout: chdir = "" 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: auth supported = cephx 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: filestore xattr use omap = true 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: mon clock drift allowed = 1.000 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: osd crush chooseleaf type = 0 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: auth debug = true 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on old message = true 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on bug = true 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.292 INFO:teuthology.orchestra.run.smithi022.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon pg warn max object skew = 0 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default size = 2 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow primary affinity = true 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow pg remap = true 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on no sortbitwise = false 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on too few osds = false 2024-03-29T10:53:48.293 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: mon_allow_pool_size_one = true 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: osd default data pool replay window = 5 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: mon allow pool delete = true 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: mon cluster log file level = debug 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: debug asserts on shutdown = true 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: mon health detail to clog = false 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: mon host = "[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.26:6789]" 2024-03-29T10:53:48.294 INFO:teuthology.orchestra.run.smithi022.stdout: lockdep = True 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: mon client directed command retry = 5 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: mon election default strategy = 1 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: ms bind msgr2 = False 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject socket failures = 5000 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: ms type = async 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout:[osd] 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: osd journal size = 100 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.295 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub load threshold = 5.0 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub max interval = 600 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd recover clone overlap = true 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd recovery max chunk = 1048576 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug shutdown = true 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug op order = true 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify stray on activate = true 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug trim objects = true 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd open classes on start = true 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug pg log writeout = true 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.296 INFO:teuthology.orchestra.run.smithi022.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: osd map max advance = 10 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: journal zero on create = true 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: filestore apply finisher threads = 3 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: bdev debug aio = true 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug misdirected ops = true 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore block size = 96636764160 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore compression algorithm = zstd 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore compression mode = aggressive 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore fsck on mount = True 2024-03-29T10:53:48.297 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore zero block detection = True 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluefs = 20 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluestore = 20 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 20 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: debug osd = 20 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: debug rocksdb = 10 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd full ratio = 0.9 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: osd class default list = * 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: osd class load list = * 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: osd client watch timeout = 120 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify cached snaps = True 2024-03-29T10:53:48.298 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify missing on start = True 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock override recovery settings = True 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: osd objectstore = bluestore 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue = debug_random 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue cut off = debug_random 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: osd shutdown pgref assert = True 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr] 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: debug mgr = 20 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:53:48.299 INFO:teuthology.orchestra.run.smithi022.stdout: mgr/telemetry/nag = false 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout:[mon] 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: debug paxos = 20 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: mon data avail warn = 5 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: mon mgr mkfs grace = 240 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd reporter subtree level = osd 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd prime pg temp = true 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:53:48.300 INFO:teuthology.orchestra.run.smithi022.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: # 1m isn't quite enough 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon min osdmap epochs = 50 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune interval = 2 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune min = 15 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon scrub interval = 300 2024-03-29T10:53:48.301 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on pool no app = False 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: paxos service trim min = 10 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout:[client] 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: rgw cache enabled = true 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable ops log = true 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable usage log = true 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 20 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: debug objecter = 20 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout: debug rados = 20 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.a] 2024-03-29T10:53:48.302 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.c] 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.b] 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout:[global] 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: chdir = "" 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: auth supported = cephx 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: filestore xattr use omap = true 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: mon clock drift allowed = 1.000 2024-03-29T10:53:48.303 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: osd crush chooseleaf type = 0 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: auth debug = true 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on old message = true 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: ms die on bug = true 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: mon pg warn max object skew = 0 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default size = 2 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.304 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow primary affinity = true 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd allow pg remap = true 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on legacy crush tunables = false 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on crush straw calc version zero = false 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on no sortbitwise = false 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on osd down out interval zero = false 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on too few osds = false 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon_allow_pool_size_one = true 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: osd default data pool replay window = 5 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.305 INFO:teuthology.orchestra.run.smithi026.stdout: mon allow pool delete = true 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: mon cluster log file level = debug 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: debug asserts on shutdown = true 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: mon health detail to clog = false 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: mon host = "[v1:172.21.15.22:6789],[v1:172.21.15.22:6790],[v1:172.21.15.26:6789]" 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: lockdep = True 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: mon client directed command retry = 5 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: mon election default strategy = 1 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: ms bind msgr2 = False 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: ms inject socket failures = 5000 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: ms type = async 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: fsid = 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.306 INFO:teuthology.orchestra.run.smithi026.stdout:[osd] 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd journal size = 100 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub load threshold = 5.0 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd scrub max interval = 600 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock profile = high_recovery_ops 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd recover clone overlap = true 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd recovery max chunk = 1048576 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug shutdown = true 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug op order = true 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify stray on activate = true 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug trim objects = true 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd open classes on start = true 2024-03-29T10:53:48.307 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug pg log writeout = true 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: osd deep scrub update digest min age = 30 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: osd map max advance = 10 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: journal zero on create = true 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: filestore ondisk finisher threads = 3 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: filestore apply finisher threads = 3 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: bdev debug aio = true 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug misdirected ops = true 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore block size = 96636764160 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression algorithm = zstd 2024-03-29T10:53:48.308 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore compression mode = aggressive 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore fsck on mount = True 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: bluestore zero block detection = True 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluefs = 20 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: debug bluestore = 20 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 20 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: debug osd = 20 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: debug rocksdb = 10 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd full ratio = 0.9 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd nearfull ratio = 0.8 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: osd class default list = * 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: osd class load list = * 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: osd client watch timeout = 120 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify cached snaps = True 2024-03-29T10:53:48.309 INFO:teuthology.orchestra.run.smithi026.stdout: osd debug verify missing on start = True 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: osd failsafe full ratio = 0.95 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: osd mclock override recovery settings = True 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: osd objectstore = bluestore 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue = debug_random 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: osd op queue cut off = debug_random 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: osd shutdown pgref assert = True 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr] 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: debug mgr = 20 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:53:48.310 INFO:teuthology.orchestra.run.smithi026.stdout: mgr/telemetry/nag = false 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout:[mon] 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: debug mon = 20 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: debug paxos = 20 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: debug auth = 20 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: mon data avail warn = 5 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: mon mgr mkfs grace = 240 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min pgs per osd = 4 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd reporter subtree level = osd 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: mon osd prime pg temp = true 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: mon reweight min bytes per osd = 10 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T10:53:48.311 INFO:teuthology.orchestra.run.smithi026.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: # 1m isn't quite enough 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon_down_mkfs_grace = 2m 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon_warn_on_filestore_osds = false 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon min osdmap epochs = 50 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune interval = 2 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune min = 15 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon osdmap full prune txsize = 2 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon scrub interval = 300 2024-03-29T10:53:48.312 INFO:teuthology.orchestra.run.smithi026.stdout: mon warn on pool no app = False 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: paxos service trim min = 10 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout:[client] 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: rgw cache enabled = true 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable ops log = true 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: rgw enable usage log = true 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: debug monc = 20 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: debug ms = 1 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: debug objecter = 20 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout: debug rados = 20 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.a] 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.c] 2024-03-29T10:53:48.313 INFO:teuthology.orchestra.run.smithi026.stdout:[mon.b] 2024-03-29T10:53:48.315 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T10:53:48.315 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T10:53:48.410 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T10:53:48.410 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:48.410 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T10:53:48.425 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:48.425 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T10:53:48.480 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T10:53:48.480 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:48.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:53:48.480 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:53:48.559 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:53:48.559 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:48.613 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T10:53:48.614 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:53:48.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T10:53:48.614 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T10:53:48.655 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:53:48.656 DEBUG:teuthology.orchestra.run.smithi026:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:48.710 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T10:53:48.710 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T10:53:48.711 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:53:48.765 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T10:53:48.768 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:53:48.837 INFO:teuthology.orchestra.run.smithi026.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T10:53:48.839 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T10:53:48.839 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T10:53:48.840 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T10:53:48.882 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T10:53:48.898 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T10:53:48.941 INFO:teuthology.orchestra.run.smithi026.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T10:53:48.957 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T10:53:48.957 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi026.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T10:53:48.958 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T10:53:48.984 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:53:48.985 INFO:tasks.ceph:role: osd.0 2024-03-29T10:53:48.985 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T10:53:48.985 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:53:49.051 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:53:49.087 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T10:53:49.087 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T10:53:49.088 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T10:53:49.160 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T10:53:49.231 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T10:53:49.297 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:53:49.298 INFO:tasks.ceph:role: osd.1 2024-03-29T10:53:49.298 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T10:53:49.298 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:53:49.363 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:53:49.363 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:53:49.364 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:53:49.395 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T10:53:49.397 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T10:53:49.397 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T10:53:49.468 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T10:53:49.541 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T10:53:49.607 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:53:49.607 INFO:tasks.ceph:role: osd.2 2024-03-29T10:53:49.608 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T10:53:49.608 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:53:49.674 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:53:49.674 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:53:49.675 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:53:49.706 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T10:53:49.708 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T10:53:49.708 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T10:53:49.780 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T10:53:49.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T10:53:49.924 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T10:53:49.924 INFO:tasks.ceph:role: osd.3 2024-03-29T10:53:49.924 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T10:53:49.924 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:53:49.995 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:53:49.995 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:53:49.995 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:53:49.996 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:53:49.996 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:53:49.996 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:53:49.996 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:53:49.996 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:53:49.996 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:53:49.996 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:53:50.023 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2024-03-29T10:53:50.025 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2024-03-29T10:53:50.026 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T10:53:50.096 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T10:53:50.168 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:50.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:50.278+0000 7f293f21f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T10:53:50.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:50.278+0000 7f293f21f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T10:53:50.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:50.278+0000 7f293f21f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:53:50.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:50.278+0000 7f293f21f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:53:50.261 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:50.278+0000 7f293f21f640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T10:53:50.262 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:50.278+0000 7f293f21f640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T10:53:53.102 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T10:53:53.133 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:53.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:53.242+0000 7f1256938640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T10:53:53.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:53.243+0000 7f1256938640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T10:53:53.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:53.243+0000 7f1256938640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:53:53.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:53.243+0000 7f1256938640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:53:53.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:53.243+0000 7f1256938640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T10:53:53.224 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:53.243+0000 7f1256938640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T10:53:56.074 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T10:53:56.105 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:56.197 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:56.216+0000 7f3df7dbb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T10:53:56.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:56.216+0000 7f3df7dbb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T10:53:56.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:56.216+0000 7f3df7dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:53:56.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:56.216+0000 7f3df7dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:53:56.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:56.216+0000 7f3df7dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T10:53:56.198 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:56.217+0000 7f3df7dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T10:53:59.039 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T10:53:59.070 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:53:59.161 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:59.180+0000 7fd992411640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T10:53:59.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:59.180+0000 7fd992411640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T10:53:59.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:59.180+0000 7fd992411640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:53:59.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:59.180+0000 7fd992411640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:53:59.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:59.180+0000 7fd992411640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T10:53:59.162 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:53:59.181+0000 7fd992411640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T10:54:02.003 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T10:54:02.034 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T10:54:02.067 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:54:02.067 INFO:tasks.ceph:role: osd.4 2024-03-29T10:54:02.067 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T10:54:02.067 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:54:02.138 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:54:02.170 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-29T10:54:02.172 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-29T10:54:02.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T10:54:02.244 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T10:54:02.316 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T10:54:02.383 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:54:02.384 INFO:tasks.ceph:role: osd.5 2024-03-29T10:54:02.384 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T10:54:02.384 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:54:02.450 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:54:02.483 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-29T10:54:02.485 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-29T10:54:02.485 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T10:54:02.566 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T10:54:02.660 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T10:54:02.732 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:54:02.732 INFO:tasks.ceph:role: osd.6 2024-03-29T10:54:02.732 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T10:54:02.732 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T10:54:02.804 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:54:02.804 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:54:02.804 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:54:02.804 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:54:02.804 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:54:02.805 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:54:02.805 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:54:02.805 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:54:02.805 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:54:02.805 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:54:02.840 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-29T10:54:02.842 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-29T10:54:02.842 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T10:54:02.918 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T10:54:02.990 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T10:54:03.060 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T10:54:03.060 INFO:tasks.ceph:role: osd.7 2024-03-29T10:54:03.060 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T10:54:03.060 DEBUG:teuthology.orchestra.run.smithi026:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: = sunit=0 swidth=0 blks 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T10:54:03.128 INFO:teuthology.orchestra.run.smithi026.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T10:54:03.160 INFO:teuthology.orchestra.run.smithi026.stdout:Discarding blocks...Done. 2024-03-29T10:54:03.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi026.front.sepia.ceph.com -o noatime 2024-03-29T10:54:03.162 DEBUG:teuthology.orchestra.run.smithi026:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T10:54:03.237 DEBUG:teuthology.orchestra.run.smithi026:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T10:54:03.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:54:03.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:03.421+0000 7fe7b9e11640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T10:54:03.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:03.421+0000 7fe7b9e11640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T10:54:03.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:03.421+0000 7fe7b9e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:54:03.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:03.421+0000 7fe7b9e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:54:03.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:03.421+0000 7fe7b9e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T10:54:03.403 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:03.421+0000 7fe7b9e11640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T10:54:06.246 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T10:54:06.277 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:54:06.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:06.390+0000 7fb2b7c2a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T10:54:06.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:06.390+0000 7fb2b7c2a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T10:54:06.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:06.390+0000 7fb2b7c2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:54:06.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:06.390+0000 7fb2b7c2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:54:06.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:06.390+0000 7fb2b7c2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T10:54:06.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:06.390+0000 7fb2b7c2a640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T10:54:09.217 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T10:54:09.248 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:54:09.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:09.358+0000 7fcbd59ac640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T10:54:09.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:09.358+0000 7fcbd59ac640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T10:54:09.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:09.358+0000 7fcbd59ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:54:09.340 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:09.358+0000 7fcbd59ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:54:09.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:09.358+0000 7fcbd59ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T10:54:09.341 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:09.359+0000 7fcbd59ac640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T10:54:12.195 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T10:54:12.227 DEBUG:teuthology.orchestra.run.smithi026:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:54:12.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:12.337+0000 7f1b0de1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T10:54:12.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:12.337+0000 7f1b0de1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T10:54:12.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:12.337+0000 7f1b0de1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:54:12.319 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:12.337+0000 7f1b0de1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:54:12.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:12.337+0000 7f1b0de1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T10:54:12.320 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:12.338+0000 7f1b0de1e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T10:54:15.172 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T10:54:15.204 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T10:54:15.204 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:15.204 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T10:54:15.236 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:15.236 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T10:54:15.307 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:15.307 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T10:54:15.378 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:15.378 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T10:54:15.447 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:15.447 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T10:54:15.518 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:15.519 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T10:54:15.550 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:15.550 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T10:54:15.620 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:15.620 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T10:54:15.689 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:15.689 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T10:54:15.757 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:15.757 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T10:54:15.825 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:15.825 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T10:54:15.839 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:15.840 DEBUG:teuthology.orchestra.run.smithi026:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T10:54:15.880 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T10:54:15.880 DEBUG:teuthology.orchestra.run.smithi022:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:54:15.883 DEBUG:teuthology.orchestra.run.smithi026:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T10:54:15.923 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.y] 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQC8nQZmcJnSLhAA3Q7hzHUMK4N5vvDEXP4mMw== 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.0] 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQC+nQZmObizEBAAwqMfrjTVPfslgbbheZdoYw== 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.1] 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDBnQZmz6iFDhAAsNk5WO7NdQHObE/OBvQo1Q== 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.2] 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDEnQZmYHfzDBAA35xCutCEbb1aqR7EeId56Q== 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.3] 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDHnQZmspTPChAA7iB+zgLy7C86gZorhA5Uog== 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.x] 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQC8nQZmsuYUMxAAUG+elDm9B3krEepx9q0kQA== 2024-03-29T10:54:15.924 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.4] 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDLnQZmv04wGRAAdDFPWPb/QhE18/5wt/TkzA== 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.5] 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDOnQZm2qdZFxAAuKSXa3aNfw3VzuGC9nag2w== 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.6] 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDRnQZmUcp2FRAA18xPt9lwNvS7lLfwIigQpA== 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.7] 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQDUnQZm/OknFBAAJzK4BqXAxf379rWAo5TyXw== 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQC8nQZmrnvCNRAAIkv0kZv/1Ccr0gc3YFDt4g== 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout:[client.1] 2024-03-29T10:54:15.925 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQC8nQZmeShSORAAqOI7iA4WX50p8DlLYyyXOw== 2024-03-29T10:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.y] 2024-03-29T10:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC8nQZmcJnSLhAA3Q7hzHUMK4N5vvDEXP4mMw== 2024-03-29T10:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.0] 2024-03-29T10:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC+nQZmObizEBAAwqMfrjTVPfslgbbheZdoYw== 2024-03-29T10:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.1] 2024-03-29T10:54:15.946 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDBnQZmz6iFDhAAsNk5WO7NdQHObE/OBvQo1Q== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.2] 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDEnQZmYHfzDBAA35xCutCEbb1aqR7EeId56Q== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.3] 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDHnQZmspTPChAA7iB+zgLy7C86gZorhA5Uog== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[mgr.x] 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC8nQZmsuYUMxAAUG+elDm9B3krEepx9q0kQA== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.4] 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDLnQZmv04wGRAAdDFPWPb/QhE18/5wt/TkzA== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.5] 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDOnQZm2qdZFxAAuKSXa3aNfw3VzuGC9nag2w== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.6] 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDRnQZmUcp2FRAA18xPt9lwNvS7lLfwIigQpA== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[osd.7] 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQDUnQZm/OknFBAAJzK4BqXAxf379rWAo5TyXw== 2024-03-29T10:54:15.947 INFO:teuthology.orchestra.run.smithi026.stdout:[client.0] 2024-03-29T10:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC8nQZmrnvCNRAAIkv0kZv/1Ccr0gc3YFDt4g== 2024-03-29T10:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout:[client.1] 2024-03-29T10:54:15.948 INFO:teuthology.orchestra.run.smithi026.stdout: key = AQC8nQZmeShSORAAqOI7iA4WX50p8DlLYyyXOw== 2024-03-29T10:54:15.949 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:54:15.969 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:54:16.041 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.043 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.094 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.096 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.147 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.150 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.200 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.203 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.254 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:54:16.256 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T10:54:16.307 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.309 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.362 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.364 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.415 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.417 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.468 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.471 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T10:54:16.522 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:54:16.524 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:54:16.578 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:54:16.580 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T10:54:16.635 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T10:54:16.635 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T10:54:16.661 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:54:16.759 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T10:54:16.785 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T10:54:16.851 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:54:16.947 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T10:54:16.974 DEBUG:teuthology.orchestra.run.smithi026:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T10:54:17.001 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T10:54:17.098 DEBUG:teuthology.orchestra.run.smithi026:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T10:54:17.125 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:54:17.127 DEBUG:teuthology.orchestra.run.smithi026:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T10:54:17.179 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T10:54:17.180 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T10:54:17.180 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T10:54:17.182 INFO:tasks.ceph.mon.a:Started 2024-03-29T10:54:17.182 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T10:54:17.182 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T10:54:17.184 INFO:tasks.ceph.mon.c:Started 2024-03-29T10:54:17.184 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T10:54:17.184 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T10:54:17.221 INFO:tasks.ceph.mon.b:Started 2024-03-29T10:54:17.222 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T10:54:17.222 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T10:54:17.222 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T10:54:17.224 INFO:tasks.ceph.mgr.y:Started 2024-03-29T10:54:17.224 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T10:54:17.225 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T10:54:17.226 INFO:tasks.ceph.mgr.x:Started 2024-03-29T10:54:17.227 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T10:54:17.227 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.348+0000 7f5545315640 10 monclient: get_monmap_and_config 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.348+0000 7f5545315640 10 monclient: build_initial_monmap 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.348+0000 7f5545315640 10 monclient: monmap: 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:54:17.348984+0000 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:54:17.348984+0000 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:17.329 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:17.330 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:17.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.348+0000 7f5545315640 1 Processor -- start 2024-03-29T10:54:17.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.348+0000 7f5545315640 1 -- start start 2024-03-29T10:54:17.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.348+0000 7f5545315640 10 monclient: init 2024-03-29T10:54:17.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:17.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:17.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 10 monclient(hunting): picked mon.noname-a con 0x7f554009ac40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:17.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 10 monclient(hunting): picked mon.noname-b con 0x7f554009c870 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 10 monclient(hunting): picked mon.noname-c con 0x7f55400b4360 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554009dec0 con 0x7f554009ac40 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554009da30 con 0x7f55400b4360 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55400b39c0 con 0x7f554009c870 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f553effd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f554009ac40 0x7f55400b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44438/0 (socket says 172.21.15.22:44438) 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f553effd640 1 -- 172.21.15.22:0/3688448103 learned_addr learned my addr 172.21.15.22:0/3688448103 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:17.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:17.349+0000 7f5545315640 10 monclient(hunting): authenticate will time out at 758.653689s 2024-03-29T10:54:20.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 10 monclient(hunting): tick 2024-03-29T10:54:20.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 10 monclient(hunting): _check_auth_tickets 2024-03-29T10:54:20.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 1 monclient(hunting): continuing hunt 2024-03-29T10:54:20.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T10:54:20.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 1 -- 172.21.15.22:0/3688448103 >> v1:172.21.15.22:6790/0 conn(0x7f554009c870 legacy=0x7f554013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:20.331 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 1 -- 172.21.15.22:0/3688448103 >> v1:172.21.15.26:6789/0 conn(0x7f55400b4360 legacy=0x7f554014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 1 -- 172.21.15.22:0/3688448103 >> v1:172.21.15.22:6789/0 conn(0x7f554009ac40 legacy=0x7f55400b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 10 monclient(hunting): picked mon.noname-c con 0x7f55200036e0 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 10 monclient(hunting): picked mon.noname-b con 0x7f5520007300 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 10 monclient(hunting): picked mon.noname-a con 0x7f554009ac40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.349+0000 7f553cff9640 1 -- 172.21.15.22:0/3688448103 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55400b5160 con 0x7f554009ac40 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.350+0000 7f553cff9640 1 -- 172.21.15.22:0/3688448103 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f552000f820 con 0x7f55200036e0 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.350+0000 7f553cff9640 1 -- 172.21.15.22:0/3688448103 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5520010a80 con 0x7f5520007300 2024-03-29T10:54:20.332 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:20.350+0000 7f553cff9640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.513+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061688730 0 0) 0x7f55400b5160 con 0x7f554009ac40 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.513+0000 7f553dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.513+0000 7f553dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.513+0000 7f553dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.513+0000 7f553dffb640 10 monclient(hunting): my global_id is 4101 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.513+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f552c003520 con 0x7f554009ac40 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3562082751 0 0) 0x7f552000f820 con 0x7f55200036e0 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:22.495 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): my global_id is 4105 2024-03-29T10:54:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55400b5160 con 0x7f55200036e0 2024-03-29T10:54:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927091230 0 0) 0x7f5520010a80 con 0x7f5520007300 2024-03-29T10:54:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient(hunting): my global_id is 4106 2024-03-29T10:54:22.496 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f552000f820 con 0x7f5520007300 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 956729760 0 0) 0x7f55400b5160 con 0x7f55200036e0 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 >> v1:172.21.15.22:6790/0 conn(0x7f5520007300 legacy=0x7f5520009730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 >> v1:172.21.15.22:6789/0 conn(0x7f554009ac40 legacy=0x7f552000cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 monclient: found mon.noname-c 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5520011ce0 con 0x7f55200036e0 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:22.498 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5538001470 con 0x7f55200036e0 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f553dffb640 10 monclient: dump: 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:22.499 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f5545315640 5 monclient: authenticate success, global_id 4105 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.514+0000 7f5545315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f55380015d0 con 0x7f55200036e0 2024-03-29T10:54:22.500 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f553dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f553dffb640 1 -- 172.21.15.22:0/3688448103 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5538001a20 con 0x7f55200036e0 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f5545315640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f5545315640 4 set_mon_vals no callback set 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f553dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f553dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.515+0000 7f553dffb640 10 monclient: dump: 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.501 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.516+0000 7f5545315640 10 monclient: shutdown 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.516+0000 7f5545315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.516+0000 7f5545315640 1 -- 172.21.15.22:0/3688448103 >> v1:172.21.15.26:6789/0 conn(0x7f55200036e0 legacy=0x7f5520005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.516+0000 7f5545315640 1 -- 172.21.15.22:0/3688448103 shutdown_connections 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.516+0000 7f5545315640 1 -- 172.21.15.22:0/3688448103 >> 172.21.15.22:0/3688448103 conn(0x7f5540066b00 msgr2=0x7f5540066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.516+0000 7f5545315640 1 -- 172.21.15.22:0/3688448103 shutdown_connections 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.516+0000 7f5545315640 1 -- 172.21.15.22:0/3688448103 wait complete. 2024-03-29T10:54:22.502 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.517+0000 7f5545315640 10 monclient: build_initial_monmap 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.517+0000 7f5545315640 10 monclient: monmap: 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.517+0000 7f5545315640 1 librados: starting msgr at 2024-03-29T10:54:22.503 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.517+0000 7f5545315640 1 librados: starting objecter 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 1 Processor -- start 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 1 -- start start 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 1 librados: setting wanted keys 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 1 librados: calling monclient init 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 10 monclient: init 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 10 monclient(hunting): picked mon.noname-c con 0x7f55200036e0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:22.509 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 10 monclient(hunting): picked mon.noname-a con 0x7f5520007300 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 10 monclient(hunting): picked mon.noname-b con 0x7f554009ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554006ddd0 con 0x7f5520007300 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.518+0000 7f5545315640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554006e040 con 0x7f554009ac40 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f5545315640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f554006e320 con 0x7f55200036e0 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f5545315640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f5545315640 10 monclient(hunting): authenticate will time out at 763.823100s 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f553effd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f55200036e0 0x7f554006cd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55524/0 (socket says 172.21.15.22:55524) 2024-03-29T10:54:22.510 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f553e7fc640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f5520007300 0x7f55401e2b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:60184/0 (socket says 172.21.15.22:60184) 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f553effd640 1 -- 172.21.15.22:0/1235156315 learned_addr learned my addr 172.21.15.22:0/1235156315 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 client.?.objecter ms_handle_connect 0x7f55200036e0 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 client.?.objecter ms_handle_connect 0x7f5520007300 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 client.?.objecter ms_handle_connect 0x7f554009ac40 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 410786407 0 0) 0x7f554006e320 con 0x7f55200036e0 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): my global_id is 4112 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5520010250 con 0x7f55200036e0 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003154055 0 0) 0x7f554006e040 con 0x7f554009ac40 2024-03-29T10:54:22.511 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 10 monclient(hunting): my global_id is 4111 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554006e320 con 0x7f554009ac40 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.519+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754233248 0 0) 0x7f5520010250 con 0x7f55200036e0 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f554006e040 con 0x7f55200036e0 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5538001d80 con 0x7f55200036e0 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): dump: 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.512 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1150971630 0 0) 0x7f554006e320 con 0x7f554009ac40 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5520010250 con 0x7f554009ac40 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5530001bf0 con 0x7f554009ac40 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): dump: 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:22.513 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185180040 0 0) 0x7f554006ddd0 con 0x7f5520007300 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): my global_id is 4107 2024-03-29T10:54:22.514 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f554006e320 con 0x7f5520007300 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1895386168 0 0) 0x7f554006e320 con 0x7f5520007300 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f554006ddd0 con 0x7f5520007300 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f55280018d0 con 0x7f5520007300 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.520+0000 7f55377fe640 10 monclient(hunting): dump: 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:22.515 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3152025913 0 0) 0x7f5520010250 con 0x7f554009ac40 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 >> v1:172.21.15.22:6790/0 conn(0x7f55200036e0 legacy=0x7f554006cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 >> v1:172.21.15.22:6789/0 conn(0x7f5520007300 legacy=0x7f55401e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 1 monclient: found mon.b 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55401369a0 con 0x7f554009ac40 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5530002000 con 0x7f554009ac40 2024-03-29T10:54:22.516 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f5545315640 5 monclient: authenticate success, global_id 4111 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f5545315640 10 monclient: _renew_subs 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f5545315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55400b7970 con 0x7f554009ac40 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f553d7fa640 4 set_mon_vals no callback set 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5530003b10 con 0x7f554009ac40 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.522+0000 7f55377fe640 10 monclient: dump: 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:22.517 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f5545315640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f5545315640 10 monclient: _renew_subs 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f5545315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5540070e10 con 0x7f554009ac40 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f5545315640 1 librados: init done 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5530004000 con 0x7f554009ac40 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2827862331 0 0) 0x7f55300054f0 con 0x7f554009ac40 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f55377fe640 10 client.4111.objecter ms_dispatch 0x7f5540000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f55377fe640 3 client.4111.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f55377fe640 3 client.4111.objecter handle_osd_map decoding full epoch 1 2024-03-29T10:54:22.518 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.523+0000 7f55377fe640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-29T10:54:22.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.524+0000 7f5545315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:22.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.524+0000 7f5545315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:22.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.524+0000 7f5545315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:22.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.524+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55400b77c0 con 0x7f554009ac40 2024-03-29T10:54:22.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.526+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f55300056b0 con 0x7f554009ac40 2024-03-29T10:54:22.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.526+0000 7f55377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:22.519 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.526+0000 7f55377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:22.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.596+0000 7f5545315640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:54:22.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.596+0000 7f5545315640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:54:22.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.596+0000 7f5545315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:22.577 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.596+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5540074bb0 con 0x7f554009ac40 2024-03-29T10:54:22.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.599+0000 7f55377fe640 1 -- 172.21.15.22:0/1235156315 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f55300042b0 con 0x7f554009ac40 2024-03-29T10:54:22.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.599+0000 7f55377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T10:54:22.580 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.599+0000 7f55377fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T10:54:22.580 INFO:teuthology.orchestra.run.smithi022.stderr:adjusted tunables profile to default 2024-03-29T10:54:22.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 10 librados: watch_flush enter 2024-03-29T10:54:22.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 10 librados: watch_flush exit 2024-03-29T10:54:22.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-29T10:54:22.581 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 10 client.4111.objecter successfully canceled tick 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 10 monclient: shutdown 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 >> v1:172.21.15.26:6789/0 conn(0x7f554009ac40 legacy=0x7f55401e6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 shutdown_connections 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 >> 172.21.15.22:0/1235156315 conn(0x7f5540066b00 msgr2=0x7f5540066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 shutdown_connections 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 1 -- 172.21.15.22:0/1235156315 wait complete. 2024-03-29T10:54:22.582 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:22.600+0000 7f5545315640 1 librados: shutdown 2024-03-29T10:54:22.592 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T10:54:22.592 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T10:54:22.592 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:22.592 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T10:54:22.619 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:22.619 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T10:54:22.686 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:22.686 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T10:54:22.753 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2024-03-29T10:54:22.753 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T10:54:22.820 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:22.820 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T10:54:22.847 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:22.847 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T10:54:22.914 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:22.914 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T10:54:22.981 DEBUG:teuthology.orchestra.run.smithi026:> set -ex 2024-03-29T10:54:22.981 DEBUG:teuthology.orchestra.run.smithi026:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T10:54:23.004 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.023+0000 7f82c8e4c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:54:23.005 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.025+0000 7f9426830040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T10:54:23.031 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new dd397594-0f34-4d00-96b5-a1f909d5cc8c 0 2024-03-29T10:54:23.060 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.079+0000 7f82c8e4c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:54:23.062 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.082+0000 7f9426830040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T10:54:23.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.182+0000 7f063051c640 10 monclient: get_monmap_and_config 2024-03-29T10:54:23.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.182+0000 7f063051c640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.162 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.182+0000 7f063051c640 10 monclient: monmap: 2024-03-29T10:54:23.162 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:23.182583+0000 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:23.182583+0000 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:23.163 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.182+0000 7f063051c640 1 Processor -- start 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.182+0000 7f063051c640 1 -- start start 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.182+0000 7f063051c640 10 monclient: init 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 10 monclient(hunting): picked mon.noname-c con 0x7f062809ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 10 monclient(hunting): picked mon.noname-a con 0x7f062809c870 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 10 monclient(hunting): picked mon.noname-b con 0x7f06280b4360 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062809dec0 con 0x7f062809c870 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062809da30 con 0x7f062809ac40 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06280b39c0 con 0x7f06280b4360 2024-03-29T10:54:23.172 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f063051c640 10 monclient(hunting): authenticate will time out at 770.538662s 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062e2b8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f062809ac40 0x7f06280b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33226/0 (socket says 172.21.15.26:33226) 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062e2b8640 1 -- 172.21.15.26:0/2915350723 learned_addr learned my addr 172.21.15.26:0/2915350723 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119525224 0 0) 0x7f062809da30 con 0x7f062809ac40 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062d2b6640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062d2b6640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062d2b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062d2b6640 10 monclient(hunting): my global_id is 4117 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.183+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0614003520 con 0x7f062809ac40 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284367849 0 0) 0x7f0624001110 con 0x7f062809c870 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.173 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): my global_id is 4113 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f062809da30 con 0x7f062809c870 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969491899 0 0) 0x7f06280b39c0 con 0x7f06280b4360 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient(hunting): my global_id is 4118 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0624001110 con 0x7f06280b4360 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1131657890 0 0) 0x7f0614003520 con 0x7f062809ac40 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 >> v1:172.21.15.22:6790/0 conn(0x7f06280b4360 legacy=0x7f062814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 >> v1:172.21.15.22:6789/0 conn(0x7f062809c870 legacy=0x7f062813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 monclient: found mon.noname-c 2024-03-29T10:54:23.174 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06280b5160 con 0x7f062809ac40 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 5 monclient: authenticate success, global_id 4117 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f061c001690 con 0x7f062809ac40 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: dump: 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.175 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f061c0018e0 con 0x7f062809ac40 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 1 -- 172.21.15.26:0/2915350723 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f061c001c70 con 0x7f062809ac40 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 4 set_mon_vals no callback set 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.176 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f062d2b6640 10 monclient: dump: 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 10 monclient: shutdown 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 1 -- 172.21.15.26:0/2915350723 >> v1:172.21.15.26:6789/0 conn(0x7f062809ac40 legacy=0x7f06280b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.177 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 1 -- 172.21.15.26:0/2915350723 shutdown_connections 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.184+0000 7f063051c640 1 -- 172.21.15.26:0/2915350723 >> 172.21.15.26:0/2915350723 conn(0x7f0628066b00 msgr2=0x7f0628066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.185+0000 7f063051c640 1 -- 172.21.15.26:0/2915350723 shutdown_connections 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.185+0000 7f063051c640 1 -- 172.21.15.26:0/2915350723 wait complete. 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.185+0000 7f063051c640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.185+0000 7f063051c640 10 monclient: monmap: 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:23.178 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.185+0000 7f063051c640 1 librados: starting msgr at 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.185+0000 7f063051c640 1 librados: starting objecter 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.185+0000 7f063051c640 1 Processor -- start 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 1 -- start start 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 1 librados: setting wanted keys 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 1 librados: calling monclient init 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient: init 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient(hunting): picked mon.noname-c con 0x7f062809ac40 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient(hunting): picked mon.noname-b con 0x7f0628142670 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient(hunting): picked mon.noname-a con 0x7f062813b520 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06280aded0 con 0x7f062813b520 2024-03-29T10:54:23.179 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06281cc340 con 0x7f0628142670 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06281cd1f0 con 0x7f062809ac40 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f063051c640 10 monclient(hunting): authenticate will time out at 770.541648s 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f062dab7640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f0628142670 0x7f06281da600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33232/0 (socket says 172.21.15.26:33232) 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f062dab7640 1 -- 172.21.15.26:0/2259753080 learned_addr learned my addr 172.21.15.26:0/2259753080 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 client.?.objecter ms_handle_connect 0x7f0628142670 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 client.?.objecter ms_handle_connect 0x7f062809ac40 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 client.?.objecter ms_handle_connect 0x7f062813b520 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274620587 0 0) 0x7f06281cc340 con 0x7f0628142670 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.180 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): my global_id is 4120 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c004eb0 con 0x7f0628142670 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 187986013 0 0) 0x7f06281cd1f0 con 0x7f062809ac40 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): my global_id is 4121 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06281cc340 con 0x7f062809ac40 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300699407 0 0) 0x7f06280aded0 con 0x7f062813b520 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 10 monclient(hunting): my global_id is 4116 2024-03-29T10:54:23.181 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.186+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06281cd1f0 con 0x7f062813b520 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482442708 0 0) 0x7f060c004eb0 con 0x7f0628142670 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06280aded0 con 0x7f0628142670 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f06240019a0 con 0x7f0628142670 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): dump: 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.182 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2496328833 0 0) 0x7f06281cc340 con 0x7f062809ac40 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f060c004eb0 con 0x7f062809ac40 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4201561108 0 0) 0x7f06281cd1f0 con 0x7f062813b520 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06281cc340 con 0x7f062813b520 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f061c001bf0 con 0x7f062809ac40 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): dump: 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.183 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0620001b50 con 0x7f062813b520 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient(hunting): dump: 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.184 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 921745166 0 0) 0x7f06280aded0 con 0x7f0628142670 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 >> v1:172.21.15.22:6790/0 conn(0x7f062809ac40 legacy=0x7f062813ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 >> v1:172.21.15.22:6789/0 conn(0x7f062813b520 legacy=0x7f06281e2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 monclient: found mon.b 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06281ccb00 con 0x7f0628142670 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.185 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 5 monclient: authenticate success, global_id 4120 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 10 monclient: _renew_subs 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06281ccdb0 con 0x7f0628142670 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 10 monclient: _renew_subs 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06281daef0 con 0x7f0628142670 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f063051c640 1 librados: init done 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0624003610 con 0x7f0628142670 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0624003920 con 0x7f0628142670 2024-03-29T10:54:23.186 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f062cab5640 4 set_mon_vals no callback set 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 10 monclient: dump: 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.187+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0624003e90 con 0x7f0628142670 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3374941869 0 0) 0x7f0624006590 con 0x7f0628142670 2024-03-29T10:54:23.187 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f061affd640 10 client.4120.objecter ms_dispatch 0x7f0628000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f061affd640 3 client.4120.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f061affd640 3 client.4120.objecter handle_osd_map decoding full epoch 2 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f061affd640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f063051c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f063051c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f063051c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.188+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0610005080 con 0x7f0628142670 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.190+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0624003780 con 0x7f0628142670 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.190+0000 7f061affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.190+0000 7f061affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.264+0000 7f063051c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "dd397594-0f34-4d00-96b5-a1f909d5cc8c", "id": 0}] 2024-03-29T10:54:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.264+0000 7f063051c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "dd397594-0f34-4d00-96b5-a1f909d5cc8c", "id": 0}] 2024-03-29T10:54:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.264+0000 7f063051c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.245 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.264+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "dd397594-0f34-4d00-96b5-a1f909d5cc8c", "id": 0} v 0) v1 -- 0x7f06100058b0 con 0x7f0628142670 2024-03-29T10:54:23.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.267+0000 7f061affd640 1 -- 172.21.15.26:0/2259753080 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dd397594-0f34-4d00-96b5-a1f909d5cc8c", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1732793696 0 732707515) 0x7f0624004470 con 0x7f0628142670 2024-03-29T10:54:23.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.267+0000 7f061affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "dd397594-0f34-4d00-96b5-a1f909d5cc8c", "id": 0}] 2024-03-29T10:54:23.248 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.267+0000 7f061affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:23.248 INFO:teuthology.orchestra.run.smithi026.stdout:0 2024-03-29T10:54:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 10 librados: watch_flush enter 2024-03-29T10:54:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 10 librados: watch_flush exit 2024-03-29T10:54:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T10:54:23.249 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 10 client.4120.objecter successfully canceled tick 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 10 monclient: shutdown 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 >> v1:172.21.15.26:6789/0 conn(0x7f0628142670 legacy=0x7f06281da600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 shutdown_connections 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 >> 172.21.15.26:0/2259753080 conn(0x7f0628066b00 msgr2=0x7f0628066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 shutdown_connections 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 1 -- 172.21.15.26:0/2259753080 wait complete. 2024-03-29T10:54:23.250 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.268+0000 7f063051c640 1 librados: shutdown 2024-03-29T10:54:23.259 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new 89d98e1f-fd1d-4578-bbe3-8e197702debd 1 2024-03-29T10:54:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.378+0000 7f77979e6640 10 monclient: get_monmap_and_config 2024-03-29T10:54:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.378+0000 7f77979e6640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.378+0000 7f77979e6640 10 monclient: monmap: 2024-03-29T10:54:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:23.379195+0000 2024-03-29T10:54:23.359 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:23.379195+0000 2024-03-29T10:54:23.360 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.360 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.360 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.360 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:23.360 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:23.360 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.379+0000 7f77979e6640 1 Processor -- start 2024-03-29T10:54:23.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.379+0000 7f77979e6640 1 -- start start 2024-03-29T10:54:23.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.379+0000 7f77979e6640 10 monclient: init 2024-03-29T10:54:23.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.368 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 10 monclient(hunting): picked mon.noname-b con 0x7f779009ac40 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 10 monclient(hunting): picked mon.noname-a con 0x7f779009c870 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 10 monclient(hunting): picked mon.noname-c con 0x7f77900b4360 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779009dec0 con 0x7f779009c870 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f779009da30 con 0x7f77900b4360 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77900b39c0 con 0x7f779009ac40 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 10 monclient(hunting): authenticate will time out at 770.735307s 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7795f83640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f77900b4360 0x7f779014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33246/0 (socket says 172.21.15.26:33246) 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7795f83640 1 -- 172.21.15.26:0/3539397607 learned_addr learned my addr 172.21.15.26:0/3539397607 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497640471 0 0) 0x7f779009da30 con 0x7f77900b4360 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.369 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): my global_id is 4123 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f777c003520 con 0x7f77900b4360 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461396203 0 0) 0x7f7780001110 con 0x7f779009c870 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): my global_id is 4119 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f779009da30 con 0x7f779009c870 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841365046 0 0) 0x7f77900b39c0 con 0x7f779009ac40 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient(hunting): my global_id is 4124 2024-03-29T10:54:23.370 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7780001110 con 0x7f779009ac40 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2909936056 0 0) 0x7f777c003520 con 0x7f77900b4360 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 >> v1:172.21.15.22:6790/0 conn(0x7f779009ac40 legacy=0x7f77900b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 >> v1:172.21.15.22:6789/0 conn(0x7f779009c870 legacy=0x7f779013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 monclient: found mon.noname-c 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77900b5160 con 0x7f77900b4360 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7788001a10 con 0x7f77900b4360 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 5 monclient: authenticate success, global_id 4123 2024-03-29T10:54:23.371 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 10 monclient: dump: 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f77979e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.380+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7788001bf0 con 0x7f77900b4360 2024-03-29T10:54:23.372 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f7787fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f7787fff640 1 -- 172.21.15.26:0/3539397607 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7788002040 con 0x7f77900b4360 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 4 set_mon_vals no callback set 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f7787fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f7787fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f7787fff640 10 monclient: dump: 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.373 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 10 monclient: shutdown 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 1 -- 172.21.15.26:0/3539397607 >> v1:172.21.15.26:6789/0 conn(0x7f77900b4360 legacy=0x7f779014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 1 -- 172.21.15.26:0/3539397607 shutdown_connections 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 1 -- 172.21.15.26:0/3539397607 >> 172.21.15.26:0/3539397607 conn(0x7f7790066b00 msgr2=0x7f7790066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 1 -- 172.21.15.26:0/3539397607 shutdown_connections 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.381+0000 7f77979e6640 1 -- 172.21.15.26:0/3539397607 wait complete. 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 10 monclient: monmap: 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:23.374 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 1 librados: starting msgr at 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 1 librados: starting objecter 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 1 Processor -- start 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 1 -- start start 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 1 librados: setting wanted keys 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 1 librados: calling monclient init 2024-03-29T10:54:23.375 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.382+0000 7f77979e6640 10 monclient: init 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 10 monclient(hunting): picked mon.noname-c con 0x7f779009ac40 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 10 monclient(hunting): picked mon.noname-b con 0x7f7790143860 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7790142cc0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77901463d0 con 0x7f7790142cc0 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77901454b0 con 0x7f7790143860 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77901d75c0 con 0x7f779009ac40 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f77979e6640 10 monclient(hunting): authenticate will time out at 770.738358s 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7794f81640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f7790143860 0x7f77901399d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33248/0 (socket says 172.21.15.26:33248) 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7794f81640 1 -- 172.21.15.26:0/2262510739 learned_addr learned my addr 172.21.15.26:0/2262510739 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 client.?.objecter ms_handle_connect 0x7f7790143860 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.376 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 client.?.objecter ms_handle_connect 0x7f7790142cc0 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 client.?.objecter ms_handle_connect 0x7f779009ac40 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864392756 0 0) 0x7f77901454b0 con 0x7f7790143860 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): my global_id is 4126 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774004eb0 con 0x7f7790143860 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046513770 0 0) 0x7f77901463d0 con 0x7f7790142cc0 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.377 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): my global_id is 4122 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77901454b0 con 0x7f7790142cc0 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069118350 0 0) 0x7f77901d75c0 con 0x7f779009ac40 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): my global_id is 4127 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77901463d0 con 0x7f779009ac40 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3843388352 0 0) 0x7f7774004eb0 con 0x7f7790143860 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77901d75c0 con 0x7f7790143860 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7780001b80 con 0x7f7790143860 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:23.378 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 758242165 0 0) 0x7f77901454b0 con 0x7f7790142cc0 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7774004eb0 con 0x7f7790142cc0 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7788001f00 con 0x7f7790142cc0 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.379 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539015524 0 0) 0x7f77901463d0 con 0x7f779009ac40 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77901454b0 con 0x7f779009ac40 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f778c0018f0 con 0x7f779009ac40 2024-03-29T10:54:23.380 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.383+0000 7f7785ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 245266184 0 0) 0x7f77901d75c0 con 0x7f7790143860 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 >> v1:172.21.15.22:6790/0 conn(0x7f779009ac40 legacy=0x7f77901d6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.381 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 >> v1:172.21.15.22:6789/0 conn(0x7f7790142cc0 legacy=0x7f779013a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 monclient: found mon.b 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779013a7f0 con 0x7f7790143860 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 5 monclient: authenticate success, global_id 4126 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 10 monclient: _renew_subs 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77901434a0 con 0x7f7790143860 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77800038a0 con 0x7f7790143860 2024-03-29T10:54:23.382 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7780003c30 con 0x7f7790143860 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 monclient: dump: 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.383 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f77800041e0 con 0x7f7790143860 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77877fe640 4 set_mon_vals no callback set 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 10 client.4126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 10 monclient: _renew_subs 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7790145db0 con 0x7f7790143860 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f77979e6640 1 librados: init done 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 555391943 0 0) 0x7f7780005bd0 con 0x7f7790143860 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 10 client.4126.objecter ms_dispatch 0x7f7790000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 3 client.4126.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 3 client.4126.objecter handle_osd_map decoding full epoch 3 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.384+0000 7f7785ffb640 20 client.4126.objecter dump_active .. 0 homeless 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.385+0000 7f77979e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.385+0000 7f77979e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.385+0000 7f77979e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.384 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.385+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779009b020 con 0x7f7790143860 2024-03-29T10:54:23.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.388+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7780005d90 con 0x7f7790143860 2024-03-29T10:54:23.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.388+0000 7f7785ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.385 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.388+0000 7f7785ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:23.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.462+0000 7f77979e6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "89d98e1f-fd1d-4578-bbe3-8e197702debd", "id": 1}] 2024-03-29T10:54:23.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.462+0000 7f77979e6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "89d98e1f-fd1d-4578-bbe3-8e197702debd", "id": 1}] 2024-03-29T10:54:23.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.462+0000 7f77979e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.462+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "89d98e1f-fd1d-4578-bbe3-8e197702debd", "id": 1} v 0) v1 -- 0x7f779013c750 con 0x7f7790143860 2024-03-29T10:54:23.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.465+0000 7f7785ffb640 1 -- 172.21.15.26:0/2262510739 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "89d98e1f-fd1d-4578-bbe3-8e197702debd", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 664320417 0 940483276) 0x7f7780005f30 con 0x7f7790143860 2024-03-29T10:54:23.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.465+0000 7f7785ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "89d98e1f-fd1d-4578-bbe3-8e197702debd", "id": 1}] 2024-03-29T10:54:23.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.465+0000 7f7785ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:23.445 INFO:teuthology.orchestra.run.smithi026.stdout:1 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.465+0000 7f77979e6640 10 librados: watch_flush enter 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.465+0000 7f77979e6640 10 librados: watch_flush exit 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.465+0000 7f77979e6640 20 client.4126.objecter shutdown clearing up homeless session... 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.465+0000 7f77979e6640 10 client.4126.objecter successfully canceled tick 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 10 monclient: shutdown 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 >> v1:172.21.15.26:6789/0 conn(0x7f7790143860 legacy=0x7f77901399d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 shutdown_connections 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 >> 172.21.15.26:0/2262510739 conn(0x7f7790066b00 msgr2=0x7f779013e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 shutdown_connections 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 1 -- 172.21.15.26:0/2262510739 wait complete. 2024-03-29T10:54:23.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.466+0000 7f77979e6640 1 librados: shutdown 2024-03-29T10:54:23.456 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new 32f00e39-6735-4e83-85f9-49fbd82a22b2 2 2024-03-29T10:54:23.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.578+0000 7ff43a0ba640 10 monclient: get_monmap_and_config 2024-03-29T10:54:23.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.578+0000 7ff43a0ba640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.558 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.578+0000 7ff43a0ba640 10 monclient: monmap: 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:23.578592+0000 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:23.578592+0000 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:23.559 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.578+0000 7ff43a0ba640 1 Processor -- start 2024-03-29T10:54:23.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.578+0000 7ff43a0ba640 1 -- start start 2024-03-29T10:54:23.568 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.578+0000 7ff43a0ba640 10 monclient: init 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff43409ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff43409c870 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff4340b4360 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43409dec0 con 0x7ff4340b4360 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff43409da30 con 0x7ff43409ac40 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4340b39c0 con 0x7ff43409c870 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff43a0ba640 10 monclient(hunting): authenticate will time out at 770.934715s 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff4337fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff43409ac40 0x7ff4340b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33256/0 (socket says 172.21.15.26:33256) 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff4337fe640 1 -- 172.21.15.26:0/2363464333 learned_addr learned my addr 172.21.15.26:0/2363464333 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1739497455 0 0) 0x7ff43409da30 con 0x7ff43409ac40 2024-03-29T10:54:23.569 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff4327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff4327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff4327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.579+0000 7ff4327fc640 10 monclient(hunting): my global_id is 4129 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff420003520 con 0x7ff43409ac40 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358324631 0 0) 0x7ff42c001490 con 0x7ff4340b4360 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): my global_id is 4125 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43409da30 con 0x7ff4340b4360 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042540906 0 0) 0x7ff4340b39c0 con 0x7ff43409c870 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.570 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient(hunting): my global_id is 4130 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff42c001490 con 0x7ff43409c870 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2825045620 0 0) 0x7ff420003520 con 0x7ff43409ac40 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 >> v1:172.21.15.22:6790/0 conn(0x7ff43409c870 legacy=0x7ff43413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 >> v1:172.21.15.22:6789/0 conn(0x7ff4340b4360 legacy=0x7ff43414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 monclient: found mon.noname-c 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4340b5160 con 0x7ff43409ac40 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff424001690 con 0x7ff43409ac40 2024-03-29T10:54:23.571 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 5 monclient: authenticate success, global_id 4129 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: dump: 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.572 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff424001870 con 0x7ff43409ac40 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 1 -- 172.21.15.26:0/2363464333 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff424001c80 con 0x7ff43409ac40 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 4 set_mon_vals no callback set 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff4327fc640 10 monclient: dump: 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.573 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 10 monclient: shutdown 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 1 -- 172.21.15.26:0/2363464333 >> v1:172.21.15.26:6789/0 conn(0x7ff43409ac40 legacy=0x7ff4340b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 1 -- 172.21.15.26:0/2363464333 shutdown_connections 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 1 -- 172.21.15.26:0/2363464333 >> 172.21.15.26:0/2363464333 conn(0x7ff434066b00 msgr2=0x7ff434066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.580+0000 7ff43a0ba640 1 -- 172.21.15.26:0/2363464333 shutdown_connections 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.581+0000 7ff43a0ba640 1 -- 172.21.15.26:0/2363464333 wait complete. 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.581+0000 7ff43a0ba640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.581+0000 7ff43a0ba640 10 monclient: monmap: 2024-03-29T10:54:23.574 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.581+0000 7ff43a0ba640 1 librados: starting msgr at 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.581+0000 7ff43a0ba640 1 librados: starting objecter 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.581+0000 7ff43a0ba640 1 Processor -- start 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 1 -- start start 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 1 librados: setting wanted keys 2024-03-29T10:54:23.575 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 1 librados: calling monclient init 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient: init 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient(hunting): picked mon.noname-c con 0x7ff43409ac40 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient(hunting): picked mon.noname-a con 0x7ff4341c88e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient(hunting): picked mon.noname-b con 0x7ff4341cacb0 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4341ca2b0 con 0x7ff4341c88e0 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4341ca520 con 0x7ff4341cacb0 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4341c85c0 con 0x7ff43409ac40 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff43a0ba640 10 monclient(hunting): authenticate will time out at 770.937717s 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff433fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ff4341cacb0 0x7ff4341e2a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33262/0 (socket says 172.21.15.26:33262) 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff433fff640 1 -- 172.21.15.26:0/690896297 learned_addr learned my addr 172.21.15.26:0/690896297 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.576 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 client.?.objecter ms_handle_connect 0x7ff4341cacb0 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 client.?.objecter ms_handle_connect 0x7ff43409ac40 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 client.?.objecter ms_handle_connect 0x7ff4341c88e0 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156276626 0 0) 0x7ff4341ca520 con 0x7ff4341cacb0 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 10 monclient(hunting): my global_id is 4132 2024-03-29T10:54:23.577 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.582+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff414004eb0 con 0x7ff4341cacb0 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051023480 0 0) 0x7ff4341c85c0 con 0x7ff43409ac40 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): my global_id is 4133 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4341ca520 con 0x7ff43409ac40 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476335459 0 0) 0x7ff4341ca2b0 con 0x7ff4341c88e0 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): my global_id is 4128 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4341c85c0 con 0x7ff4341c88e0 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 206745081 0 0) 0x7ff414004eb0 con 0x7ff4341cacb0 2024-03-29T10:54:23.578 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4341ca2b0 con 0x7ff4341cacb0 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff42c001bf0 con 0x7ff4341cacb0 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): dump: 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.579 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2600000396 0 0) 0x7ff4341ca520 con 0x7ff43409ac40 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff414004eb0 con 0x7ff43409ac40 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff424001b80 con 0x7ff43409ac40 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): dump: 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.580 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 850518538 0 0) 0x7ff4341c85c0 con 0x7ff4341c88e0 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4341ca520 con 0x7ff4341c88e0 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff41c0017a0 con 0x7ff4341c88e0 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient(hunting): dump: 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.581 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3355631123 0 0) 0x7ff4341ca2b0 con 0x7ff4341cacb0 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 >> v1:172.21.15.22:6790/0 conn(0x7ff43409ac40 legacy=0x7ff4341d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 >> v1:172.21.15.22:6789/0 conn(0x7ff4341c88e0 legacy=0x7ff4340b1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 monclient: found mon.b 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4340b1710 con 0x7ff4341cacb0 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.582 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 5 monclient: authenticate success, global_id 4132 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 10 monclient: _renew_subs 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4340b1960 con 0x7ff4341cacb0 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 10 monclient: _renew_subs 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff434142d50 con 0x7ff4341cacb0 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff43a0ba640 1 librados: init done 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff42c0020c0 con 0x7ff4341cacb0 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff42c003b30 con 0x7ff4341cacb0 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 monclient: dump: 2024-03-29T10:54:23.583 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff431ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff42c004020 con 0x7ff4341cacb0 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2911423515 0 0) 0x7ff42c0042d0 con 0x7ff4341cacb0 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 10 client.4132.objecter ms_dispatch 0x7ff434000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 3 client.4132.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T10:54:23.584 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 3 client.4132.objecter handle_osd_map decoding full epoch 4 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.583+0000 7ff42bfff640 20 client.4132.objecter dump_active .. 0 homeless 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.584+0000 7ff43a0ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.584+0000 7ff43a0ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.584+0000 7ff43a0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.584+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff418005080 con 0x7ff4341cacb0 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.586+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff42c0046c0 con 0x7ff4341cacb0 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.586+0000 7ff42bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.585 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.586+0000 7ff42bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:23.611 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.631+0000 7f82c8e4c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:54:23.638 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.657+0000 7f9426830040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T10:54:23.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.660+0000 7ff43a0ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "32f00e39-6735-4e83-85f9-49fbd82a22b2", "id": 2}] 2024-03-29T10:54:23.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.660+0000 7ff43a0ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "32f00e39-6735-4e83-85f9-49fbd82a22b2", "id": 2}] 2024-03-29T10:54:23.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.660+0000 7ff43a0ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.660+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "32f00e39-6735-4e83-85f9-49fbd82a22b2", "id": 2} v 0) v1 -- 0x7ff4180058b0 con 0x7ff4341cacb0 2024-03-29T10:54:23.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.663+0000 7ff42bfff640 1 -- 172.21.15.26:0/690896297 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "32f00e39-6735-4e83-85f9-49fbd82a22b2", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2716022020 0 216599125) 0x7ff42c005c10 con 0x7ff4341cacb0 2024-03-29T10:54:23.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.663+0000 7ff42bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "32f00e39-6735-4e83-85f9-49fbd82a22b2", "id": 2}] 2024-03-29T10:54:23.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.663+0000 7ff42bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:23.644 INFO:teuthology.orchestra.run.smithi026.stdout:2 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.663+0000 7ff43a0ba640 10 librados: watch_flush enter 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.663+0000 7ff43a0ba640 10 librados: watch_flush exit 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.663+0000 7ff43a0ba640 20 client.4132.objecter shutdown clearing up homeless session... 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.663+0000 7ff43a0ba640 10 client.4132.objecter successfully canceled tick 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 10 monclient: shutdown 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 >> v1:172.21.15.26:6789/0 conn(0x7ff4341cacb0 legacy=0x7ff4341e2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 shutdown_connections 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 >> 172.21.15.26:0/690896297 conn(0x7ff434066b00 msgr2=0x7ff4340b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 shutdown_connections 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 1 -- 172.21.15.26:0/690896297 wait complete. 2024-03-29T10:54:23.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.664+0000 7ff43a0ba640 1 librados: shutdown 2024-03-29T10:54:23.654 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new 30bbbc18-5aa5-4e64-bb3f-192e4c68fae9 3 2024-03-29T10:54:23.668 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.688+0000 7f82c8e4c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:54:23.696 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.716+0000 7f9426830040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T10:54:23.715 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.734+0000 7f82c8e4c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:54:23.744 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.763+0000 7f9426830040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.774+0000 7f27d44f6640 10 monclient: get_monmap_and_config 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.774+0000 7f27d44f6640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.774+0000 7f27d44f6640 10 monclient: monmap: 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:23.775299+0000 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:23.775299+0000 2024-03-29T10:54:23.755 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.756 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.756 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.756 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:23.756 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:23.756 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.775+0000 7f27d44f6640 1 Processor -- start 2024-03-29T10:54:23.763 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.775+0000 7f27d44f6640 1 -- start start 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.775+0000 7f27d44f6640 10 monclient: init 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 10 monclient(hunting): picked mon.noname-a con 0x7f27cc09ac40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 10 monclient(hunting): picked mon.noname-b con 0x7f27cc09c870 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 10 monclient(hunting): picked mon.noname-c con 0x7f27cc0b4360 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc09dec0 con 0x7f27cc09ac40 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc09da30 con 0x7f27cc0b4360 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc0b39c0 con 0x7f27cc09c870 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 10 monclient(hunting): authenticate will time out at 771.131302s 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d2a93640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f27cc0b4360 0x7f27cc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33266/0 (socket says 172.21.15.26:33266) 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d2a93640 1 -- 172.21.15.26:0/762190863 learned_addr learned my addr 172.21.15.26:0/762190863 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.764 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536674222 0 0) 0x7f27cc09da30 con 0x7f27cc0b4360 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): my global_id is 4135 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b8003520 con 0x7f27cc0b4360 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283762417 0 0) 0x7f27c0001110 con 0x7f27cc09ac40 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): my global_id is 4131 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc09da30 con 0x7f27cc09ac40 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843184906 0 0) 0x7f27cc0b39c0 con 0x7f27cc09c870 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.765 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient(hunting): my global_id is 4136 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c0001110 con 0x7f27cc09c870 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 801519128 0 0) 0x7f27b8003520 con 0x7f27cc0b4360 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 >> v1:172.21.15.22:6790/0 conn(0x7f27cc09c870 legacy=0x7f27cc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 >> v1:172.21.15.22:6789/0 conn(0x7f27cc09ac40 legacy=0x7f27cc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 monclient: found mon.noname-c 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0b5160 con 0x7f27cc0b4360 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.766 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c4001a10 con 0x7f27cc0b4360 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d1290640 10 monclient: dump: 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 5 monclient: authenticate success, global_id 4135 2024-03-29T10:54:23.767 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.776+0000 7f27d44f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27c4001b70 con 0x7f27cc0b4360 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d1290640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d1290640 1 -- 172.21.15.26:0/762190863 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c4001fc0 con 0x7f27cc0b4360 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d1290640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d1290640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d1290640 10 monclient: dump: 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.768 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 4 set_mon_vals no callback set 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 10 monclient: shutdown 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 1 -- 172.21.15.26:0/762190863 >> v1:172.21.15.26:6789/0 conn(0x7f27cc0b4360 legacy=0x7f27cc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 1 -- 172.21.15.26:0/762190863 shutdown_connections 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 1 -- 172.21.15.26:0/762190863 >> 172.21.15.26:0/762190863 conn(0x7f27cc066b00 msgr2=0x7f27cc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 1 -- 172.21.15.26:0/762190863 shutdown_connections 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.777+0000 7f27d44f6640 1 -- 172.21.15.26:0/762190863 wait complete. 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 10 monclient: monmap: 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:23.769 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 1 librados: starting msgr at 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 1 librados: starting objecter 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 1 Processor -- start 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 1 -- start start 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 1 librados: setting wanted keys 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 1 librados: calling monclient init 2024-03-29T10:54:23.770 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.778+0000 7f27d44f6640 10 monclient: init 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient(hunting): picked mon.noname-b con 0x7f27cc09ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient(hunting): picked mon.noname-a con 0x7f27cc0af6a0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient(hunting): picked mon.noname-c con 0x7f27cc1df670 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc1db010 con 0x7f27cc0af6a0 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc1da100 con 0x7f27cc09ac40 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc1da3e0 con 0x7f27cc1df670 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient(hunting): authenticate will time out at 771.134227s 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d2292640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f27cc09ac40 0x7f27cc1da900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33270/0 (socket says 172.21.15.26:33270) 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d2292640 1 -- 172.21.15.26:0/3840465787 learned_addr learned my addr 172.21.15.26:0/3840465787 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 client.?.objecter ms_handle_connect 0x7f27cc09ac40 2024-03-29T10:54:23.771 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 client.?.objecter ms_handle_connect 0x7f27cc0af6a0 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 client.?.objecter ms_handle_connect 0x7f27cc1df670 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238037164 0 0) 0x7f27cc1da100 con 0x7f27cc09ac40 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): my global_id is 4138 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b0004eb0 con 0x7f27cc09ac40 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3563196362 0 0) 0x7f27cc1db010 con 0x7f27cc0af6a0 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.772 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): my global_id is 4134 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc1da100 con 0x7f27cc0af6a0 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085948652 0 0) 0x7f27cc1da3e0 con 0x7f27cc1df670 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): my global_id is 4139 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc1db010 con 0x7f27cc1df670 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2469565528 0 0) 0x7f27b0004eb0 con 0x7f27cc09ac40 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27cc1da3e0 con 0x7f27cc09ac40 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c0001520 con 0x7f27cc09ac40 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): dump: 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.773 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2244214365 0 0) 0x7f27cc1da100 con 0x7f27cc0af6a0 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27b0004eb0 con 0x7f27cc0af6a0 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c8001870 con 0x7f27cc0af6a0 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.774 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): dump: 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1628859711 0 0) 0x7f27cc1db010 con 0x7f27cc1df670 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27cc1da100 con 0x7f27cc1df670 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c4001e80 con 0x7f27cc1df670 2024-03-29T10:54:23.775 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient(hunting): dump: 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 223505296 0 0) 0x7f27cc1da3e0 con 0x7f27cc09ac40 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 >> v1:172.21.15.22:6790/0 conn(0x7f27cc1df670 legacy=0x7f27cc1e1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.776 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 >> v1:172.21.15.22:6789/0 conn(0x7f27cc0af6a0 legacy=0x7f27cc140fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 monclient: found mon.b 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc140400 con 0x7f27cc09ac40 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 5 monclient: authenticate success, global_id 4138 2024-03-29T10:54:23.777 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient: _renew_subs 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.779+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc1416b0 con 0x7f27cc09ac40 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 10 monclient: _renew_subs 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc1e3360 con 0x7f27cc09ac40 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 1 librados: init done 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27c00038a0 con 0x7f27cc09ac40 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c0003c70 con 0x7f27cc09ac40 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d0a8f640 4 set_mon_vals no callback set 2024-03-29T10:54:23.778 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 10 monclient: dump: 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f27c0004220 con 0x7f27cc09ac40 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 274690768 0 0) 0x7f27c0006870 con 0x7f27cc09ac40 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 10 client.4138.objecter ms_dispatch 0x7f27cc000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 3 client.4138.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T10:54:23.779 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 3 client.4138.objecter handle_osd_map decoding full epoch 5 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27beffd640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.780+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b4005080 con 0x7f27cc09ac40 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.782+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f27c0006360 con 0x7f27cc09ac40 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.782+0000 7f27beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:23.780 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.782+0000 7f27beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:23.780 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.790+0000 7f82c8e4c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:54:23.801 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.821+0000 7f9426830040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T10:54:23.815 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.835+0000 7f82c8e4c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:54:23.836 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.856+0000 7f27d44f6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "30bbbc18-5aa5-4e64-bb3f-192e4c68fae9", "id": 3}] 2024-03-29T10:54:23.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.856+0000 7f27d44f6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "30bbbc18-5aa5-4e64-bb3f-192e4c68fae9", "id": 3}] 2024-03-29T10:54:23.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.856+0000 7f27d44f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:23.837 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.856+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "30bbbc18-5aa5-4e64-bb3f-192e4c68fae9", "id": 3} v 0) v1 -- 0x7f27b40058b0 con 0x7f27cc09ac40 2024-03-29T10:54:23.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.859+0000 7f27beffd640 1 -- 172.21.15.26:0/3840465787 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "30bbbc18-5aa5-4e64-bb3f-192e4c68fae9", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3365938329 0 525046306) 0x7f27c00066b0 con 0x7f27cc09ac40 2024-03-29T10:54:23.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.859+0000 7f27beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "30bbbc18-5aa5-4e64-bb3f-192e4c68fae9", "id": 3}] 2024-03-29T10:54:23.839 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.859+0000 7f27beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:23.839 INFO:teuthology.orchestra.run.smithi026.stdout:3 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.859+0000 7f27d44f6640 10 librados: watch_flush enter 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.859+0000 7f27d44f6640 10 librados: watch_flush exit 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.859+0000 7f27d44f6640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.859+0000 7f27d44f6640 10 client.4138.objecter successfully canceled tick 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 10 monclient: shutdown 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 >> v1:172.21.15.26:6789/0 conn(0x7f27cc09ac40 legacy=0x7f27cc1da900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 shutdown_connections 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 >> 172.21.15.26:0/3840465787 conn(0x7f27cc066b00 msgr2=0x7f27cc0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 shutdown_connections 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 1 -- 172.21.15.26:0/3840465787 wait complete. 2024-03-29T10:54:23.841 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:23.860+0000 7f27d44f6640 1 librados: shutdown 2024-03-29T10:54:23.846 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.866+0000 7f9426830040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T10:54:23.851 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new 0d705184-b655-464a-b001-22221eac4ddb 4 2024-03-29T10:54:23.936 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:23.955+0000 7f82c8e4c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:54:23.970 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:23.989+0000 7f9426830040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T10:54:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.010+0000 7fc52853d640 10 monclient: get_monmap_and_config 2024-03-29T10:54:23.991 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.010+0000 7fc52853d640 10 monclient: build_initial_monmap 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.010+0000 7fc52853d640 10 monclient: monmap: 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:24.011322+0000 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:24.011322+0000 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.011+0000 7fc52853d640 1 Processor -- start 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.011+0000 7fc52853d640 1 -- start start 2024-03-29T10:54:23.992 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.011+0000 7fc52853d640 10 monclient: init 2024-03-29T10:54:23.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:23.999 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient(hunting): picked mon.noname-b con 0x7fc52009ac40 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient(hunting): picked mon.noname-c con 0x7fc52009c870 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5200b4360 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52009dec0 con 0x7fc5200b4360 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc52009da30 con 0x7fc52009c870 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5200b39c0 con 0x7fc52009ac40 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient(hunting): authenticate will time out at 771.367312s 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc525ad8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc52009c870 0x7fc52013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33274/0 (socket says 172.21.15.26:33274) 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc525ad8640 1 -- 172.21.15.26:0/296683556 learned_addr learned my addr 172.21.15.26:0/296683556 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808217998 0 0) 0x7fc52009da30 con 0x7fc52009c870 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.000 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): my global_id is 4141 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc50c003520 con 0x7fc52009c870 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 957412513 0 0) 0x7fc518001490 con 0x7fc5200b4360 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): my global_id is 4137 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc52009da30 con 0x7fc5200b4360 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133007338 0 0) 0x7fc5200b39c0 con 0x7fc52009ac40 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient(hunting): my global_id is 4142 2024-03-29T10:54:24.001 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc518001490 con 0x7fc52009ac40 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1089977050 0 0) 0x7fc50c003520 con 0x7fc52009c870 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 >> v1:172.21.15.22:6790/0 conn(0x7fc52009ac40 legacy=0x7fc5200b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 >> v1:172.21.15.22:6789/0 conn(0x7fc5200b4360 legacy=0x7fc52014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 monclient: found mon.noname-c 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5200b5160 con 0x7fc52009c870 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.002 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc514001690 con 0x7fc52009c870 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 5 monclient: authenticate success, global_id 4141 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: dump: 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.003 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc514001870 con 0x7fc52009c870 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc5252d7640 1 -- 172.21.15.26:0/296683556 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc514001c80 con 0x7fc52009c870 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.012+0000 7fc52853d640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 4 set_mon_vals no callback set 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc5252d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc5252d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc5252d7640 10 monclient: dump: 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.004 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 10 monclient: shutdown 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 1 -- 172.21.15.26:0/296683556 >> v1:172.21.15.26:6789/0 conn(0x7fc52009c870 legacy=0x7fc52013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 1 -- 172.21.15.26:0/296683556 shutdown_connections 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 1 -- 172.21.15.26:0/296683556 >> 172.21.15.26:0/296683556 conn(0x7fc520066b00 msgr2=0x7fc520066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 1 -- 172.21.15.26:0/296683556 shutdown_connections 2024-03-29T10:54:24.005 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.013+0000 7fc52853d640 1 -- 172.21.15.26:0/296683556 wait complete. 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 10 monclient: monmap: 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 1 librados: starting msgr at 2024-03-29T10:54:24.006 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 1 librados: starting objecter 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 1 Processor -- start 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 1 -- start start 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 1 librados: setting wanted keys 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 1 librados: calling monclient init 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.014+0000 7fc52853d640 10 monclient: init 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient(hunting): picked mon.noname-a con 0x7fc52009ac40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5200b12b0 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5201414d0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5201db2c0 con 0x7fc52009ac40 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc520141960 con 0x7fc5200b12b0 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5201dadc0 con 0x7fc5201414d0 2024-03-29T10:54:24.007 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient(hunting): authenticate will time out at 771.370256s 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc525ad8640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc5200b12b0 0x7fc520140dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33282/0 (socket says 172.21.15.26:33282) 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc525ad8640 1 -- 172.21.15.26:0/4258398594 learned_addr learned my addr 172.21.15.26:0/4258398594 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 client.?.objecter ms_handle_connect 0x7fc5200b12b0 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 client.?.objecter ms_handle_connect 0x7fc52009ac40 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 client.?.objecter ms_handle_connect 0x7fc5201414d0 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538330822 0 0) 0x7fc520141960 con 0x7fc5200b12b0 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): my global_id is 4144 2024-03-29T10:54:24.008 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc504004eb0 con 0x7fc5200b12b0 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351356914 0 0) 0x7fc5201db2c0 con 0x7fc52009ac40 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): my global_id is 4140 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc520141960 con 0x7fc52009ac40 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1660522437 0 0) 0x7fc5201dadc0 con 0x7fc5201414d0 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): my global_id is 4145 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5201db2c0 con 0x7fc5201414d0 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 958917508 0 0) 0x7fc504004eb0 con 0x7fc5200b12b0 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5201dadc0 con 0x7fc5200b12b0 2024-03-29T10:54:24.009 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc514001bc0 con 0x7fc5200b12b0 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:24.010 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3519791719 0 0) 0x7fc520141960 con 0x7fc52009ac40 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc504004eb0 con 0x7fc52009ac40 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc51c001870 con 0x7fc52009ac40 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.011 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1486367468 0 0) 0x7fc5201db2c0 con 0x7fc5201414d0 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc520141960 con 0x7fc5201414d0 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc518001a00 con 0x7fc5201414d0 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.012 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1564364195 0 0) 0x7fc5201dadc0 con 0x7fc5200b12b0 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 >> v1:172.21.15.22:6790/0 conn(0x7fc5201414d0 legacy=0x7fc5201e1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 >> v1:172.21.15.22:6789/0 conn(0x7fc52009ac40 legacy=0x7fc5201da6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 monclient: found mon.b 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5201daf50 con 0x7fc5200b12b0 2024-03-29T10:54:24.013 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc512ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 5 monclient: authenticate success, global_id 4144 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient: _renew_subs 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.015+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5201e30a0 con 0x7fc5200b12b0 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 10 monclient: _renew_subs 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5201e3410 con 0x7fc5200b12b0 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 1 librados: init done 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc514004bf0 con 0x7fc5200b12b0 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.014 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc514004f00 con 0x7fc5200b12b0 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc524ad6640 4 set_mon_vals no callback set 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 10 monclient: dump: 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.015 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc5140053f0 con 0x7fc5200b12b0 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 3619564308 0 0) 0x7fc514005bb0 con 0x7fc5200b12b0 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 10 client.4144.objecter ms_dispatch 0x7fc520000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc512ffd640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.016+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc508005080 con 0x7fc5200b12b0 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.018+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc514005f10 con 0x7fc5200b12b0 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.018+0000 7fc512ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.016 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.018+0000 7fc512ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:24.017 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.028+0000 7f82c8e4c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:54:24.046 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.064+0000 7f9426830040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T10:54:24.057 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.076+0000 7f82c8e4c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:54:24.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.092+0000 7fc52853d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0d705184-b655-464a-b001-22221eac4ddb", "id": 4}] 2024-03-29T10:54:24.072 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.092+0000 7fc52853d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0d705184-b655-464a-b001-22221eac4ddb", "id": 4}] 2024-03-29T10:54:24.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.092+0000 7fc52853d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.073 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.092+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0d705184-b655-464a-b001-22221eac4ddb", "id": 4} v 0) v1 -- 0x7fc5080058b0 con 0x7fc5200b12b0 2024-03-29T10:54:24.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.095+0000 7fc512ffd640 1 -- 172.21.15.26:0/4258398594 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0d705184-b655-464a-b001-22221eac4ddb", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2677195110 0 1697012583) 0x7fc5140056a0 con 0x7fc5200b12b0 2024-03-29T10:54:24.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.095+0000 7fc512ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0d705184-b655-464a-b001-22221eac4ddb", "id": 4}] 2024-03-29T10:54:24.075 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.095+0000 7fc512ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:24.075 INFO:teuthology.orchestra.run.smithi026.stdout:4 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.095+0000 7fc52853d640 10 librados: watch_flush enter 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.095+0000 7fc52853d640 10 librados: watch_flush exit 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.095+0000 7fc52853d640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.095+0000 7fc52853d640 10 client.4144.objecter successfully canceled tick 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 10 monclient: shutdown 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 >> v1:172.21.15.26:6789/0 conn(0x7fc5200b12b0 legacy=0x7fc520140dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 shutdown_connections 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 >> 172.21.15.26:0/4258398594 conn(0x7fc520066b00 msgr2=0x7fc52013cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 shutdown_connections 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 1 -- 172.21.15.26:0/4258398594 wait complete. 2024-03-29T10:54:24.077 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.096+0000 7fc52853d640 1 librados: shutdown 2024-03-29T10:54:24.086 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new 04210bd3-557d-4820-ba3f-2ff1a251cdaa 5 2024-03-29T10:54:24.094 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.114+0000 7f9426830040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T10:54:24.112 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.131+0000 7f82c8e4c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:54:24.151 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.171+0000 7f9426830040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T10:54:24.161 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.180+0000 7f82c8e4c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.207+0000 7fc872c4e640 10 monclient: get_monmap_and_config 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.207+0000 7fc872c4e640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.207+0000 7fc872c4e640 10 monclient: monmap: 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:24.207632+0000 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:24.207632+0000 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.188 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:24.189 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:24.189 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.207+0000 7fc872c4e640 1 Processor -- start 2024-03-29T10:54:24.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.207+0000 7fc872c4e640 1 -- start start 2024-03-29T10:54:24.189 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.207+0000 7fc872c4e640 10 monclient: init 2024-03-29T10:54:24.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.196 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c09ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c09c870 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c0b4360 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c09dec0 con 0x7fc86c0b4360 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c09da30 con 0x7fc86c09ac40 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0b39c0 con 0x7fc86c09c870 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc872c4e640 10 monclient(hunting): authenticate will time out at 771.563643s 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc8709ea640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc86c09ac40 0x7fc86c0b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33284/0 (socket says 172.21.15.26:33284) 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc8709ea640 1 -- 172.21.15.26:0/3924480857 learned_addr learned my addr 172.21.15.26:0/3924480857 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2238412273 0 0) 0x7fc86c09da30 con 0x7fc86c09ac40 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.197 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc86b7fe640 10 monclient(hunting): my global_id is 4147 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.208+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc85c003520 con 0x7fc86c09ac40 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436202546 0 0) 0x7fc86c0b39c0 con 0x7fc86c09c870 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): my global_id is 4148 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c09da30 con 0x7fc86c09c870 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721773810 0 0) 0x7fc864001490 con 0x7fc86c0b4360 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.198 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient(hunting): my global_id is 4143 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c0b39c0 con 0x7fc86c0b4360 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 974210961 0 0) 0x7fc85c003520 con 0x7fc86c09ac40 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 >> v1:172.21.15.22:6790/0 conn(0x7fc86c09c870 legacy=0x7fc86c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 >> v1:172.21.15.22:6789/0 conn(0x7fc86c0b4360 legacy=0x7fc86c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 monclient: found mon.noname-c 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0b5160 con 0x7fc86c09ac40 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 5 monclient: authenticate success, global_id 4147 2024-03-29T10:54:24.199 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc860001690 con 0x7fc86c09ac40 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: dump: 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.200 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc860001870 con 0x7fc86c09ac40 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 1 -- 172.21.15.26:0/3924480857 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc860001c80 con 0x7fc86c09ac40 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 4 set_mon_vals no callback set 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc86b7fe640 10 monclient: dump: 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.201 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 10 monclient: shutdown 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 1 -- 172.21.15.26:0/3924480857 >> v1:172.21.15.26:6789/0 conn(0x7fc86c09ac40 legacy=0x7fc86c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 1 -- 172.21.15.26:0/3924480857 shutdown_connections 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 1 -- 172.21.15.26:0/3924480857 >> 172.21.15.26:0/3924480857 conn(0x7fc86c066b00 msgr2=0x7fc86c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.209+0000 7fc872c4e640 1 -- 172.21.15.26:0/3924480857 shutdown_connections 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 1 -- 172.21.15.26:0/3924480857 wait complete. 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 10 monclient: monmap: 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:24.202 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 1 librados: starting msgr at 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 1 librados: starting objecter 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 1 Processor -- start 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 1 -- start start 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 1 librados: setting wanted keys 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 1 librados: calling monclient init 2024-03-29T10:54:24.203 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.210+0000 7fc872c4e640 10 monclient: init 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 10 monclient(hunting): picked mon.noname-b con 0x7fc86c09ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 10 monclient(hunting): picked mon.noname-a con 0x7fc86c1d7220 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 10 monclient(hunting): picked mon.noname-c con 0x7fc86c1dfb50 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c142eb0 con 0x7fc86c1d7220 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0b7700 con 0x7fc86c09ac40 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc86c0b7970 con 0x7fc86c1dfb50 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc872c4e640 10 monclient(hunting): authenticate will time out at 771.566711s 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8709ea640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fc86c09ac40 0x7fc86c0b8100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33290/0 (socket says 172.21.15.26:33290) 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8709ea640 1 -- 172.21.15.26:0/2437198859 learned_addr learned my addr 172.21.15.26:0/2437198859 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 client.?.objecter ms_handle_connect 0x7fc86c09ac40 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 client.?.objecter ms_handle_connect 0x7fc86c1d7220 2024-03-29T10:54:24.204 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 client.?.objecter ms_handle_connect 0x7fc86c1dfb50 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2363639007 0 0) 0x7fc86c0b7700 con 0x7fc86c09ac40 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 10 monclient(hunting): my global_id is 4150 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.211+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc850004eb0 con 0x7fc86c09ac40 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2822258978 0 0) 0x7fc86c142eb0 con 0x7fc86c1d7220 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): my global_id is 4146 2024-03-29T10:54:24.205 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c0b7700 con 0x7fc86c1d7220 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1378392114 0 0) 0x7fc86c0b7970 con 0x7fc86c1dfb50 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): my global_id is 4151 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc86c142eb0 con 0x7fc86c1dfb50 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 381094519 0 0) 0x7fc850004eb0 con 0x7fc86c09ac40 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc86c0b7970 con 0x7fc86c09ac40 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc860001c40 con 0x7fc86c09ac40 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): dump: 2024-03-29T10:54:24.206 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2084128605 0 0) 0x7fc86c0b7700 con 0x7fc86c1d7220 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc850004eb0 con 0x7fc86c1d7220 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2205913074 0 0) 0x7fc86c142eb0 con 0x7fc86c1dfb50 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc86c0b7700 con 0x7fc86c1dfb50 2024-03-29T10:54:24.207 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3190694959 0 0) 0x7fc86c0b7970 con 0x7fc86c09ac40 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 >> v1:172.21.15.22:6790/0 conn(0x7fc86c1dfb50 legacy=0x7fc86c1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 >> v1:172.21.15.22:6789/0 conn(0x7fc86c1d7220 legacy=0x7fc86c1d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 monclient: found mon.b 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc86c0b8810 con 0x7fc86c09ac40 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 5 monclient: authenticate success, global_id 4150 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 10 monclient: _renew_subs 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86c0b1a30 con 0x7fc86c09ac40 2024-03-29T10:54:24.208 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 10 monclient: _renew_subs 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc86c1d7f30 con 0x7fc86c09ac40 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc872c4e640 1 librados: init done 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc860001170 con 0x7fc86c09ac40 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc860004d90 con 0x7fc86c09ac40 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc86affd640 4 set_mon_vals no callback set 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 monclient: dump: 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.209 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc860005280 con 0x7fc86c09ac40 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1963397594 0 0) 0x7fc860005ab0 con 0x7fc86c09ac40 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 10 client.4150.objecter ms_dispatch 0x7fc86c000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.212+0000 7fc8697fa640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.213+0000 7fc872c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.210 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.213+0000 7fc872c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.213+0000 7fc872c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.213+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c005080 con 0x7fc86c09ac40 2024-03-29T10:54:24.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.215+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc86c1d7f30 con 0x7fc86c09ac40 2024-03-29T10:54:24.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.215+0000 7fc8697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.211 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.215+0000 7fc8697fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:24.211 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.222+0000 7f9426830040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T10:54:24.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.289+0000 7fc872c4e640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "04210bd3-557d-4820-ba3f-2ff1a251cdaa", "id": 5}] 2024-03-29T10:54:24.269 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.289+0000 7fc872c4e640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "04210bd3-557d-4820-ba3f-2ff1a251cdaa", "id": 5}] 2024-03-29T10:54:24.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.289+0000 7fc872c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.270 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.289+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd new", "uuid": "04210bd3-557d-4820-ba3f-2ff1a251cdaa", "id": 5} v 0) v1 -- 0x7fc84c0058b0 con 0x7fc86c09ac40 2024-03-29T10:54:24.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.292+0000 7fc8697fa640 1 -- 172.21.15.26:0/2437198859 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04210bd3-557d-4820-ba3f-2ff1a251cdaa", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1911687837 0 1988412176) 0x7fc860005dc0 con 0x7fc86c09ac40 2024-03-29T10:54:24.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.292+0000 7fc8697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "04210bd3-557d-4820-ba3f-2ff1a251cdaa", "id": 5}] 2024-03-29T10:54:24.272 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.292+0000 7fc8697fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:24.273 INFO:teuthology.orchestra.run.smithi026.stdout:5 2024-03-29T10:54:24.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 10 librados: watch_flush enter 2024-03-29T10:54:24.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 10 librados: watch_flush exit 2024-03-29T10:54:24.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T10:54:24.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 10 client.4150.objecter successfully canceled tick 2024-03-29T10:54:24.274 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 10 monclient: shutdown 2024-03-29T10:54:24.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 >> v1:172.21.15.26:6789/0 conn(0x7fc86c09ac40 legacy=0x7fc86c0b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 shutdown_connections 2024-03-29T10:54:24.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 >> 172.21.15.26:0/2437198859 conn(0x7fc86c066b00 msgr2=0x7fc86c13c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 shutdown_connections 2024-03-29T10:54:24.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 1 -- 172.21.15.26:0/2437198859 wait complete. 2024-03-29T10:54:24.275 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.293+0000 7fc872c4e640 1 librados: shutdown 2024-03-29T10:54:24.283 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new 608227e6-bd7d-40be-bfe3-d7205697aef8 6 2024-03-29T10:54:24.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.443+0000 7fe1e20c8640 10 monclient: get_monmap_and_config 2024-03-29T10:54:24.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.443+0000 7fe1e20c8640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.424 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.443+0000 7fe1e20c8640 10 monclient: monmap: 2024-03-29T10:54:24.424 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:24.424 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.424 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:24.444048+0000 2024-03-29T10:54:24.425 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:24.444048+0000 2024-03-29T10:54:24.425 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.425 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.425 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.425 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:24.425 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:24.425 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.444+0000 7fe1e20c8640 1 Processor -- start 2024-03-29T10:54:24.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.444+0000 7fe1e20c8640 1 -- start start 2024-03-29T10:54:24.435 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.444+0000 7fe1e20c8640 10 monclient: init 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.444+0000 7fe1e20c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.444+0000 7fe1e20c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.444+0000 7fe1e20c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe1dc09ac40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.444+0000 7fe1e20c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe1dc09c870 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe1dc0b4360 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc09dec0 con 0x7fe1dc09ac40 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc09da30 con 0x7fe1dc0b4360 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc0b39c0 con 0x7fe1dc09c870 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 10 monclient(hunting): authenticate will time out at 771.800135s 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1dbfff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe1dc0b4360 0x7fe1dc14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33304/0 (socket says 172.21.15.26:33304) 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1dbfff640 1 -- 172.21.15.26:0/3945074808 learned_addr learned my addr 172.21.15.26:0/3945074808 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1770893677 0 0) 0x7fe1dc09da30 con 0x7fe1dc0b4360 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.436 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 4153 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c8003520 con 0x7fe1dc0b4360 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836877470 0 0) 0x7fe1c4001110 con 0x7fe1dc09ac40 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 4149 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc09da30 con 0x7fe1dc09ac40 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168420824 0 0) 0x7fe1dc0b39c0 con 0x7fe1dc09c870 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient(hunting): my global_id is 4154 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1c4001110 con 0x7fe1dc09c870 2024-03-29T10:54:24.437 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2541949585 0 0) 0x7fe1c8003520 con 0x7fe1dc0b4360 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 >> v1:172.21.15.22:6790/0 conn(0x7fe1dc09c870 legacy=0x7fe1dc13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 >> v1:172.21.15.22:6789/0 conn(0x7fe1dc09ac40 legacy=0x7fe1dc0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 monclient: found mon.noname-c 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc0b5160 con 0x7fe1dc0b4360 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 5 monclient: authenticate success, global_id 4153 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe1d4001a10 con 0x7fe1dc0b4360 2024-03-29T10:54:24.438 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: dump: 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1d4001b70 con 0x7fe1dc0b4360 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.439 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 1 -- 172.21.15.26:0/3945074808 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe1d4001f80 con 0x7fe1dc0b4360 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1e20c8640 4 set_mon_vals no callback set 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.445+0000 7fe1da7fc640 10 monclient: dump: 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.440 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.446+0000 7fe1e20c8640 10 monclient: shutdown 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.446+0000 7fe1e20c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.446+0000 7fe1e20c8640 1 -- 172.21.15.26:0/3945074808 >> v1:172.21.15.26:6789/0 conn(0x7fe1dc0b4360 legacy=0x7fe1dc14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.446+0000 7fe1e20c8640 1 -- 172.21.15.26:0/3945074808 shutdown_connections 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.446+0000 7fe1e20c8640 1 -- 172.21.15.26:0/3945074808 >> 172.21.15.26:0/3945074808 conn(0x7fe1dc066b00 msgr2=0x7fe1dc066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.446+0000 7fe1e20c8640 1 -- 172.21.15.26:0/3945074808 shutdown_connections 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.446+0000 7fe1e20c8640 1 -- 172.21.15.26:0/3945074808 wait complete. 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 10 monclient: monmap: 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:24.441 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 1 librados: starting msgr at 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 1 librados: starting objecter 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 1 Processor -- start 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 1 -- start start 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 1 librados: setting wanted keys 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 1 librados: calling monclient init 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.447+0000 7fe1e20c8640 10 monclient: init 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:24.442 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe1dc09ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe1dc1d71e0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe1dc139520 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc1426a0 con 0x7fe1dc139520 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc1d6d00 con 0x7fe1dc09ac40 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc142980 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1e20c8640 10 monclient(hunting): authenticate will time out at 771.803421s 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1db7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe1dc09ac40 0x7fe1dc140d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33316/0 (socket says 172.21.15.26:33316) 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1db7fe640 1 -- 172.21.15.26:0/683389667 learned_addr learned my addr 172.21.15.26:0/683389667 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 client.?.objecter ms_handle_connect 0x7fe1dc09ac40 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 client.?.objecter ms_handle_connect 0x7fe1dc139520 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 client.?.objecter ms_handle_connect 0x7fe1dc1d71e0 2024-03-29T10:54:24.443 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1462797189 0 0) 0x7fe1dc1d6d00 con 0x7fe1dc09ac40 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): my global_id is 4156 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc004eb0 con 0x7fe1dc09ac40 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906684478 0 0) 0x7fe1dc1426a0 con 0x7fe1dc139520 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): my global_id is 4152 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc1d6d00 con 0x7fe1dc139520 2024-03-29T10:54:24.444 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181944884 0 0) 0x7fe1dc142980 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): my global_id is 4157 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1dc1426a0 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 991460165 0 0) 0x7fe1dc1426a0 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc142980 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe1cc001740 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): dump: 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.445 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.448+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636375395 0 0) 0x7fe1dc1d6d00 con 0x7fe1dc139520 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc1426a0 con 0x7fe1dc139520 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2633457008 0 0) 0x7fe1bc004eb0 con 0x7fe1dc09ac40 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1dc1d6d00 con 0x7fe1dc09ac40 2024-03-29T10:54:24.446 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe1d4001e80 con 0x7fe1dc139520 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient(hunting): dump: 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe1c4001a40 con 0x7fe1dc09ac40 2024-03-29T10:54:24.447 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient(hunting): dump: 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2688964910 0 0) 0x7fe1dc142980 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.448 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 >> v1:172.21.15.26:6789/0 conn(0x7fe1dc09ac40 legacy=0x7fe1dc140d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 >> v1:172.21.15.22:6789/0 conn(0x7fe1dc139520 legacy=0x7fe1dc1e24f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 monclient: found mon.c 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc140320 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.449 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 5 monclient: authenticate success, global_id 4157 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 10 monclient: _renew_subs 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc141480 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 10 monclient: _renew_subs 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc0af8b0 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1e20c8640 1 librados: init done 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1cc001cd0 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe1cc003790 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:24.450 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d9ffb640 4 set_mon_vals no callback set 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 monclient: dump: 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe1cc003d00 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3874023413 0 0) 0x7fe1cc005570 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 10 client.4157.objecter ms_dispatch 0x7fe1dc000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T10:54:24.451 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 3 client.4157.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 3 client.4157.objecter handle_osd_map decoding full epoch 8 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.449+0000 7fe1d3fff640 20 client.4157.objecter dump_active .. 0 homeless 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.450+0000 7fe1e20c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.450+0000 7fe1e20c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.450+0000 7fe1e20c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.450+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1c0005080 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.452+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe1cc004190 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.452+0000 7fe1d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.452 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.452+0000 7fe1d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:24.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.526+0000 7fe1e20c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "608227e6-bd7d-40be-bfe3-d7205697aef8", "id": 6}] 2024-03-29T10:54:24.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.526+0000 7fe1e20c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "608227e6-bd7d-40be-bfe3-d7205697aef8", "id": 6}] 2024-03-29T10:54:24.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.526+0000 7fe1e20c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.507 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.526+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd new", "uuid": "608227e6-bd7d-40be-bfe3-d7205697aef8", "id": 6} v 0) v1 -- 0x7fe1c00058b0 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.529+0000 7fe1d3fff640 1 -- 172.21.15.26:0/683389667 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "608227e6-bd7d-40be-bfe3-d7205697aef8", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 268884927 0 1113811849) 0x7fe1cc0043b0 con 0x7fe1dc1d71e0 2024-03-29T10:54:24.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.529+0000 7fe1d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "608227e6-bd7d-40be-bfe3-d7205697aef8", "id": 6}] 2024-03-29T10:54:24.510 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.529+0000 7fe1d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:24.510 INFO:teuthology.orchestra.run.smithi026.stdout:6 2024-03-29T10:54:24.511 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 10 librados: watch_flush enter 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 10 librados: watch_flush exit 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 20 client.4157.objecter shutdown clearing up homeless session... 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 10 client.4157.objecter successfully canceled tick 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 10 monclient: shutdown 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 >> v1:172.21.15.22:6790/0 conn(0x7fe1dc1d71e0 legacy=0x7fe1dc138e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 shutdown_connections 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 >> 172.21.15.26:0/683389667 conn(0x7fe1dc066b00 msgr2=0x7fe1dc13a230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 shutdown_connections 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 1 -- 172.21.15.26:0/683389667 wait complete. 2024-03-29T10:54:24.512 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.530+0000 7fe1e20c8640 1 librados: shutdown 2024-03-29T10:54:24.521 DEBUG:teuthology.orchestra.run.smithi026:> sudo ceph --cluster ceph osd new b1d4ed43-f342-4655-91a5-33de40a75723 7 2024-03-29T10:54:24.562 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.581+0000 7f82c8e4c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:54:24.607 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.627+0000 7f9426830040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T10:54:24.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.641+0000 7f6450e4a640 10 monclient: get_monmap_and_config 2024-03-29T10:54:24.621 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.641+0000 7f6450e4a640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.641+0000 7f6450e4a640 10 monclient: monmap: 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:54:24.641624+0000 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:54:24.641624+0000 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:24.622 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.641+0000 7f6450e4a640 1 Processor -- start 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.641+0000 7f6450e4a640 1 -- start start 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.641+0000 7f6450e4a640 10 monclient: init 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f644c09ac40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f644c09c870 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f644c0b4360 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c09dec0 con 0x7f644c09ac40 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c09da30 con 0x7f644c0b4360 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c0b39c0 con 0x7f644c09c870 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.633 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6450e4a640 10 monclient(hunting): authenticate will time out at 771.997673s 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f644ad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f644c0b4360 0x7f644c14d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33330/0 (socket says 172.21.15.26:33330) 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f644ad9d640 1 -- 172.21.15.26:0/2992347051 learned_addr learned my addr 172.21.15.26:0/2992347051 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.642+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2989518408 0 0) 0x7f643c001040 con 0x7f644c09ac40 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): my global_id is 4155 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6438003520 con 0x7f644c09ac40 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1686897395 0 0) 0x7f644c0b39c0 con 0x7f644c09c870 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.634 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): my global_id is 4160 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f643c001040 con 0x7f644c09c870 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3640494977 0 0) 0x7f644c09da30 con 0x7f644c0b4360 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient(hunting): my global_id is 4159 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644c0b39c0 con 0x7f644c0b4360 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045059301 0 0) 0x7f6438003520 con 0x7f644c09ac40 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 >> v1:172.21.15.22:6790/0 conn(0x7f644c09c870 legacy=0x7f644c13f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 >> v1:172.21.15.26:6789/0 conn(0x7f644c0b4360 legacy=0x7f644c14d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 monclient: found mon.noname-a 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:54:24.635 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0b5160 con 0x7f644c09ac40 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f643c0015b0 con 0x7f644c09ac40 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: dump: 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.636 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6450e4a640 5 monclient: authenticate success, global_id 4155 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6450e4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f643c001710 con 0x7f644c09ac40 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 1 -- 172.21.15.26:0/2992347051 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f643c001b60 con 0x7f644c09ac40 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6449d9b640 10 monclient: dump: 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.637 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6450e4a640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.643+0000 7f6450e4a640 4 set_mon_vals no callback set 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.644+0000 7f6450e4a640 10 monclient: shutdown 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.644+0000 7f6450e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.644+0000 7f6450e4a640 1 -- 172.21.15.26:0/2992347051 >> v1:172.21.15.22:6789/0 conn(0x7f644c09ac40 legacy=0x7f644c0b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.644+0000 7f6450e4a640 1 -- 172.21.15.26:0/2992347051 shutdown_connections 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.644+0000 7f6450e4a640 1 -- 172.21.15.26:0/2992347051 >> 172.21.15.26:0/2992347051 conn(0x7f644c066b00 msgr2=0x7f644c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.644+0000 7f6450e4a640 1 -- 172.21.15.26:0/2992347051 shutdown_connections 2024-03-29T10:54:24.638 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.644+0000 7f6450e4a640 1 -- 172.21.15.26:0/2992347051 wait complete. 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 10 monclient: monmap: 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 0 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 0.000000 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:created 0.000000 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 1 librados: starting msgr at 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 1 librados: starting objecter 2024-03-29T10:54:24.639 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 1 Processor -- start 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 1 -- start start 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 1 librados: setting wanted keys 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 1 librados: calling monclient init 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.645+0000 7f6450e4a640 10 monclient: init 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 10 monclient(hunting): picked mon.noname-c con 0x7f644c09ac40 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 10 monclient(hunting): picked mon.noname-a con 0x7f644c1462f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 10 monclient(hunting): picked mon.noname-b con 0x7f644c0b1830 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c145c80 con 0x7f644c1462f0 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c144c40 con 0x7f644c0b1830 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c145ed0 con 0x7f644c09ac40 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f6450e4a640 10 monclient(hunting): authenticate will time out at 772.001969s 2024-03-29T10:54:24.640 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f644ad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f644c0b1830 0x7f644c1daeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.26:33338/0 (socket says 172.21.15.26:33338) 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.646+0000 7f644ad9d640 1 -- 172.21.15.26:0/2268989555 learned_addr learned my addr 172.21.15.26:0/2268989555 (peer_addr_for_me v1:172.21.15.26:0/0) 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 client.?.objecter ms_handle_connect 0x7f644c0b1830 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 client.?.objecter ms_handle_connect 0x7f644c09ac40 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 client.?.objecter ms_handle_connect 0x7f644c1462f0 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977457684 0 0) 0x7f644c144c40 con 0x7f644c0b1830 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): my global_id is 4162 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f642c004eb0 con 0x7f644c0b1830 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2794221985 0 0) 0x7f644c145ed0 con 0x7f644c09ac40 2024-03-29T10:54:24.641 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): my global_id is 4163 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644c144c40 con 0x7f644c09ac40 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1472258948 0 0) 0x7f644c145c80 con 0x7f644c1462f0 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): my global_id is 4158 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f644c145ed0 con 0x7f644c1462f0 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17770327 0 0) 0x7f644c144c40 con 0x7f644c09ac40 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f644c145c80 con 0x7f644c09ac40 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522383181 0 0) 0x7f642c004eb0 con 0x7f644c0b1830 2024-03-29T10:54:24.642 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f644c144c40 con 0x7f644c0b1830 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f643c0010d0 con 0x7f644c09ac40 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.643 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1562142016 0 0) 0x7f644c145ed0 con 0x7f644c1462f0 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f642c004eb0 con 0x7f644c1462f0 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6444001b70 con 0x7f644c0b1830 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.644 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6434001830 con 0x7f644c1462f0 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.645 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2471358953 0 0) 0x7f644c145c80 con 0x7f644c09ac40 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 >> v1:172.21.15.26:6789/0 conn(0x7f644c0b1830 legacy=0x7f644c1daeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 >> v1:172.21.15.22:6789/0 conn(0x7f644c1462f0 legacy=0x7f644c1da7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 monclient: found mon.c 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c1445f0 con 0x7f644c09ac40 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 5 monclient: authenticate success, global_id 4163 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 10 monclient: _renew_subs 2024-03-29T10:54:24.646 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f644c144840 con 0x7f644c09ac40 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 10 client.4163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 10 monclient: _renew_subs 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c143c50 con 0x7f644c09ac40 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6450e4a640 1 librados: init done 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f643c0012d0 con 0x7f644c09ac40 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f643c003a40 con 0x7f644c09ac40 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f644959a640 4 set_mon_vals no callback set 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 monclient: dump: 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:epoch 1 2024-03-29T10:54:24.647 INFO:teuthology.orchestra.run.smithi026.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:election_strategy: 1 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f643c003f70 con 0x7f644c09ac40 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 676494784 0 0) 0x7f643c005850 con 0x7f644c09ac40 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.647+0000 7f6442ffd640 10 client.4163.objecter ms_dispatch 0x7f644c000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.648+0000 7f6442ffd640 3 client.4163.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.648+0000 7f6442ffd640 3 client.4163.objecter handle_osd_map decoding full epoch 9 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.648+0000 7f6442ffd640 20 client.4163.objecter dump_active .. 0 homeless 2024-03-29T10:54:24.648 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.648+0000 7f6450e4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.648+0000 7f6450e4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.648+0000 7f6450e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.648+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6430005080 con 0x7f644c09ac40 2024-03-29T10:54:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.650+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f643c005ae0 con 0x7f644c09ac40 2024-03-29T10:54:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.650+0000 7f6442ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.649 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.650+0000 7f6442ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:24.704 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.723+0000 7f82c8e4c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:54:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.724+0000 7f6450e4a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b1d4ed43-f342-4655-91a5-33de40a75723", "id": 7}] 2024-03-29T10:54:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.724+0000 7f6450e4a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b1d4ed43-f342-4655-91a5-33de40a75723", "id": 7}] 2024-03-29T10:54:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.724+0000 7f6450e4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.705 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.724+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd new", "uuid": "b1d4ed43-f342-4655-91a5-33de40a75723", "id": 7} v 0) v1 -- 0x7f64300058b0 con 0x7f644c09ac40 2024-03-29T10:54:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6442ffd640 1 -- 172.21.15.26:0/2268989555 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b1d4ed43-f342-4655-91a5-33de40a75723", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3496724236 0 1371665406) 0x7f643c005370 con 0x7f644c09ac40 2024-03-29T10:54:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6442ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b1d4ed43-f342-4655-91a5-33de40a75723", "id": 7}] 2024-03-29T10:54:24.707 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6442ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:24.707 INFO:teuthology.orchestra.run.smithi026.stdout:7 2024-03-29T10:54:24.708 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 10 librados: watch_flush enter 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 10 librados: watch_flush exit 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 20 client.4163.objecter shutdown clearing up homeless session... 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 10 client.4163.objecter successfully canceled tick 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 10 monclient: shutdown 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 >> v1:172.21.15.22:6790/0 conn(0x7f644c09ac40 legacy=0x7f644c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 shutdown_connections 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.727+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 >> 172.21.15.26:0/2268989555 conn(0x7f644c066b00 msgr2=0x7f644c13af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.728+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 shutdown_connections 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.728+0000 7f6450e4a640 1 -- 172.21.15.26:0/2268989555 wait complete. 2024-03-29T10:54:24.709 INFO:teuthology.orchestra.run.smithi026.stderr:2024-03-29T10:54:24.728+0000 7f6450e4a640 1 librados: shutdown 2024-03-29T10:54:24.718 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T10:54:24.718 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T10:54:24.720 INFO:tasks.ceph.osd.0:Started 2024-03-29T10:54:24.720 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T10:54:24.721 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T10:54:24.722 INFO:tasks.ceph.osd.1:Started 2024-03-29T10:54:24.722 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T10:54:24.723 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T10:54:24.724 INFO:tasks.ceph.osd.2:Started 2024-03-29T10:54:24.725 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T10:54:24.725 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T10:54:24.727 INFO:tasks.ceph.osd.3:Started 2024-03-29T10:54:24.727 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T10:54:24.727 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T10:54:24.729 INFO:tasks.ceph.osd.4:Started 2024-03-29T10:54:24.729 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T10:54:24.729 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T10:54:24.731 INFO:tasks.ceph.osd.5:Started 2024-03-29T10:54:24.731 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T10:54:24.731 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T10:54:24.733 INFO:tasks.ceph.osd.6:Started 2024-03-29T10:54:24.733 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T10:54:24.733 DEBUG:teuthology.orchestra.run.smithi026:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T10:54:24.735 INFO:tasks.ceph.osd.7:Started 2024-03-29T10:54:24.735 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:24.756 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.776+0000 7f9426830040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T10:54:24.857 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.876+0000 7f82c8e4c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.887+0000 7f756bfef640 10 monclient: get_monmap_and_config 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.887+0000 7f756bfef640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.887+0000 7f756bfef640 10 monclient: monmap: 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:54:24.888038+0000 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:54:24.888038+0000 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:24.868 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.887+0000 7f756bfef640 1 Processor -- start 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 1 -- start start 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient: init 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient(hunting): picked mon.noname-c con 0x7f756409ac40 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient(hunting): picked mon.noname-a con 0x7f756409c870 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient(hunting): picked mon.noname-b con 0x7f75640b4360 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756409dec0 con 0x7f756409c870 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f756409da30 con 0x7f756409ac40 2024-03-29T10:54:24.877 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640b39c0 con 0x7f75640b4360 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756bfef640 10 monclient(hunting): authenticate will time out at 766.192814s 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756a58c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f75640b4360 0x7f756414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55594/0 (socket says 172.21.15.22:55594) 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.888+0000 7f756a58c640 1 -- 172.21.15.22:0/4239643250 learned_addr learned my addr 172.21.15.22:0/4239643250 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1634961237 0 0) 0x7f75640b39c0 con 0x7f75640b4360 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): my global_id is 4190 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7550003520 con 0x7f75640b4360 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262617253 0 0) 0x7f756409da30 con 0x7f756409ac40 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.878 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): my global_id is 4189 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75640b39c0 con 0x7f756409ac40 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432041617 0 0) 0x7f7560001110 con 0x7f756409c870 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient(hunting): my global_id is 4185 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f756409da30 con 0x7f756409c870 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 9070220 0 0) 0x7f7550003520 con 0x7f75640b4360 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 >> v1:172.21.15.26:6789/0 conn(0x7f756409ac40 legacy=0x7f75640b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 >> v1:172.21.15.22:6789/0 conn(0x7f756409c870 legacy=0x7f756413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.879 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 monclient: found mon.noname-b 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75640b5160 con 0x7f75640b4360 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f756bfef640 5 monclient: authenticate success, global_id 4190 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f755c001a10 con 0x7f75640b4360 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f756bfef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: dump: 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:24.880 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f756bfef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f755c001bf0 con 0x7f75640b4360 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f756bfef640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f756bfef640 4 set_mon_vals no callback set 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 1 -- 172.21.15.22:0/4239643250 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f755c002040 con 0x7f75640b4360 2024-03-29T10:54:24.881 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.889+0000 7f7568d89640 10 monclient: dump: 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:24.882 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 10 monclient: shutdown 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 1 -- 172.21.15.22:0/4239643250 >> v1:172.21.15.22:6790/0 conn(0x7f75640b4360 legacy=0x7f756414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 1 -- 172.21.15.22:0/4239643250 shutdown_connections 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 1 -- 172.21.15.22:0/4239643250 >> 172.21.15.22:0/4239643250 conn(0x7f7564066b00 msgr2=0x7f7564066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 1 -- 172.21.15.22:0/4239643250 shutdown_connections 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 1 -- 172.21.15.22:0/4239643250 wait complete. 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 10 monclient: build_initial_monmap 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.890+0000 7f756bfef640 10 monclient: monmap: 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2024-03-29T10:54:24.883 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 1 librados: starting msgr at 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 1 librados: starting objecter 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 1 Processor -- start 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 1 -- start start 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 1 librados: setting wanted keys 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 1 librados: calling monclient init 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 10 monclient: init 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:24.884 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 10 monclient(hunting): picked mon.noname-a con 0x7f756409ac40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 10 monclient(hunting): picked mon.noname-c con 0x7f75640aee40 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f756bfef640 10 monclient(hunting): picked mon.noname-b con 0x7f75640ae850 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f7569d8b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f756409ac40 0x7f75641d7ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:60238/0 (socket says 172.21.15.22:60238) 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.891+0000 7f7569d8b640 1 -- 172.21.15.22:0/4001632262 learned_addr learned my addr 172.21.15.22:0/4001632262 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7564134820 con 0x7f756409ac40 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 client.?.objecter ms_handle_connect 0x7f756409ac40 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640afe50 con 0x7f75640ae850 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75640adad0 con 0x7f75640aee40 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 client.?.objecter ms_handle_connect 0x7f75640aee40 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 10 monclient(hunting): authenticate will time out at 766.196263s 2024-03-29T10:54:24.885 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 client.?.objecter ms_handle_connect 0x7f75640ae850 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251843160 0 0) 0x7f7564134820 con 0x7f756409ac40 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): my global_id is 4188 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7548004eb0 con 0x7f756409ac40 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195184184 0 0) 0x7f75640adad0 con 0x7f75640aee40 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): my global_id is 4193 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7564134820 con 0x7f75640aee40 2024-03-29T10:54:24.886 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584591068 0 0) 0x7f75640afe50 con 0x7f75640ae850 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): my global_id is 4192 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75640adad0 con 0x7f75640ae850 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990903355 0 0) 0x7f7548004eb0 con 0x7f756409ac40 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75640afe50 con 0x7f756409ac40 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3788205990 0 0) 0x7f7564134820 con 0x7f75640aee40 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7548004eb0 con 0x7f75640aee40 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7554001b30 con 0x7f756409ac40 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): dump: 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:24.887 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7560001860 con 0x7f75640aee40 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient(hunting): dump: 2024-03-29T10:54:24.888 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3609373634 0 0) 0x7f75640afe50 con 0x7f756409ac40 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 >> v1:172.21.15.22:6790/0 conn(0x7f75640aee40 legacy=0x7f7564133a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 >> v1:172.21.15.26:6789/0 conn(0x7f75640ae850 legacy=0x7f7564134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 monclient: found mon.a 2024-03-29T10:54:24.889 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75641d79b0 con 0x7f756409ac40 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f755a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 5 monclient: authenticate success, global_id 4188 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 10 monclient: _renew_subs 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75641d7640 con 0x7f756409ac40 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 10 client.4188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 10 monclient: _renew_subs 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75640b07a0 con 0x7f756409ac40 2024-03-29T10:54:24.890 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.892+0000 7f756bfef640 1 librados: init done 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7554001280 con 0x7f756409ac40 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7554001c90 con 0x7f756409ac40 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 10 monclient: dump: 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:24.891 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755bfff640 4 set_mon_vals no callback set 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7554003e90 con 0x7f756409ac40 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f75540068e0 con 0x7f756409ac40 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 10 client.4188.objecter ms_dispatch 0x7f7564000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 3 client.4188.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 3 client.4188.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f755a7fc640 20 client.4188.objecter dump_active .. 0 homeless 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f756bfef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f756bfef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f756bfef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:24.892 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.893+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f754c005080 con 0x7f756409ac40 2024-03-29T10:54:24.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.895+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7554006aa0 con 0x7f756409ac40 2024-03-29T10:54:24.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.895+0000 7f755a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:24.893 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.895+0000 7f755a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:24.894 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.914+0000 7f9426830040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T10:54:24.906 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:24.925+0000 7f82c8e4c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:54:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.958+0000 7f756bfef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.958+0000 7f756bfef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.958+0000 7f756bfef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:24.939 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.958+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f754c005330 con 0x7f756409ac40 2024-03-29T10:54:24.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f755a7fc640 1 -- 172.21.15.22:0/4001632262 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f75540061d0 con 0x7f756409ac40 2024-03-29T10:54:24.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f755a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:24.940 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f755a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:24.940 INFO:teuthology.orchestra.run.smithi022.stdout: 2024-03-29T10:54:24.940 INFO:teuthology.orchestra.run.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:24.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 10 librados: watch_flush enter 2024-03-29T10:54:24.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 10 librados: watch_flush exit 2024-03-29T10:54:24.941 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 20 client.4188.objecter shutdown clearing up homeless session... 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 10 client.4188.objecter successfully canceled tick 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 10 monclient: shutdown 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 >> v1:172.21.15.22:6789/0 conn(0x7f756409ac40 legacy=0x7f75641d7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.959+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 shutdown_connections 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.960+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 >> 172.21.15.22:0/4001632262 conn(0x7f7564066b00 msgr2=0x7f756413b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.960+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 shutdown_connections 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.960+0000 7f756bfef640 1 -- 172.21.15.22:0/4001632262 wait complete. 2024-03-29T10:54:24.942 INFO:teuthology.orchestra.run.smithi022.stderr:2024-03-29T10:54:24.960+0000 7f756bfef640 1 librados: shutdown 2024-03-29T10:54:24.942 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:24.961+0000 7f9426830040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T10:54:24.950 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T10:54:24.951 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T10:54:25.024 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:25.043+0000 7f82c8e4c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:54:25.058 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:25.077+0000 7f9426830040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:54:25.125 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:25.143+0000 7f82c8e4c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:54:25.156 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:25.176+0000 7f9426830040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T10:54:25.284 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:25.304+0000 7f82c8e4c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:54:25.317 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:25.336+0000 7f9426830040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T10:54:25.328 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:25.348+0000 7f82c8e4c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:54:25.330 INFO:tasks.ceph.osd.6.smithi026.stderr:2024-03-29T10:54:25.350+0000 7f53af6a6640 -1 Falling back to public interface 2024-03-29T10:54:25.332 INFO:tasks.ceph.osd.2.smithi022.stderr:2024-03-29T10:54:25.351+0000 7f744b01a640 -1 Falling back to public interface 2024-03-29T10:54:25.334 INFO:tasks.ceph.osd.5.smithi026.stderr:2024-03-29T10:54:25.353+0000 7fbbcd40d640 -1 Falling back to public interface 2024-03-29T10:54:25.340 INFO:tasks.ceph.osd.0.smithi022.stderr:2024-03-29T10:54:25.359+0000 7f7a58e14640 -1 Falling back to public interface 2024-03-29T10:54:25.340 INFO:tasks.ceph.osd.3.smithi022.stderr:2024-03-29T10:54:25.359+0000 7fed6f8d5640 -1 Falling back to public interface 2024-03-29T10:54:25.341 INFO:tasks.ceph.osd.1.smithi022.stderr:2024-03-29T10:54:25.360+0000 7f409820f640 -1 Falling back to public interface 2024-03-29T10:54:25.341 INFO:tasks.ceph.osd.3.smithi022.stderr:2024-03-29T10:54:25.360+0000 7fed6e884640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6813/0: (98) Address already in use 2024-03-29T10:54:25.349 INFO:tasks.ceph.osd.7.smithi026.stderr:2024-03-29T10:54:25.369+0000 7f9b2d613640 -1 Falling back to public interface 2024-03-29T10:54:25.355 INFO:tasks.ceph.osd.4.smithi026.stderr:2024-03-29T10:54:25.375+0000 7fbbc0319640 -1 Falling back to public interface 2024-03-29T10:54:25.364 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:25.383+0000 7f9426830040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T10:54:25.493 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:25.512+0000 7f82c8e4c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:54:25.524 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:25.544+0000 7f9426830040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T10:54:25.757 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:25.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.865+0000 7f518c9f4640 10 monclient: get_monmap_and_config 2024-03-29T10:54:25.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.865+0000 7f518c9f4640 10 monclient: build_initial_monmap 2024-03-29T10:54:25.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.865+0000 7f518c9f4640 10 monclient: monmap: 2024-03-29T10:54:25.846 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:54:25.866233+0000 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:54:25.866233+0000 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.865+0000 7f518c9f4640 1 Processor -- start 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 1 -- start start 2024-03-29T10:54:25.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient: init 2024-03-29T10:54:25.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:25.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient(hunting): picked mon.noname-b con 0x7f51881457f0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient(hunting): picked mon.noname-a con 0x7f518813f650 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient(hunting): picked mon.noname-c con 0x7f51880a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51880aaef0 con 0x7f518813f650 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188145bd0 con 0x7f51880a9d70 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188145eb0 con 0x7f51881457f0 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518c9f4640 10 monclient(hunting): authenticate will time out at 767.170926s 2024-03-29T10:54:25.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518659c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f51881457f0 0x7f518813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55624/0 (socket says 172.21.15.22:55624) 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f5185d9b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f518813f650 0x7f51880b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:60246/0 (socket says 172.21.15.22:60246) 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.866+0000 7f518659c640 1 -- 172.21.15.22:0/1610101322 learned_addr learned my addr 172.21.15.22:0/1610101322 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130043849 0 0) 0x7f5188145eb0 con 0x7f51881457f0 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): my global_id is 4196 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5170003520 con 0x7f51881457f0 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728249577 0 0) 0x7f51880aaef0 con 0x7f518813f650 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): my global_id is 4191 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5188145eb0 con 0x7f518813f650 2024-03-29T10:54:25.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457222966 0 0) 0x7f5188145bd0 con 0x7f51880a9d70 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient(hunting): my global_id is 4195 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51880aaef0 con 0x7f51880a9d70 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410113487 0 0) 0x7f5170003520 con 0x7f51881457f0 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 >> v1:172.21.15.26:6789/0 conn(0x7f51880a9d70 legacy=0x7f518814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 >> v1:172.21.15.22:6789/0 conn(0x7f518813f650 legacy=0x7f51880b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: _finish_hunting 0 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 monclient: found mon.noname-b 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188146340 con 0x7f51881457f0 2024-03-29T10:54:25.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: _finish_auth 0 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: _check_auth_tickets 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f51780015c0 con 0x7f51881457f0 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518c9f4640 5 monclient: authenticate success, global_id 4196 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518c9f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: dump: 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.859 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518c9f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5178001720 con 0x7f51881457f0 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 1 -- 172.21.15.22:0/1610101322 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5178001b70 con 0x7f51881457f0 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518559a640 10 monclient: dump: 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.860 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518c9f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.867+0000 7f518c9f4640 4 set_mon_vals no callback set 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.868+0000 7f518c9f4640 10 monclient: shutdown 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.868+0000 7f518c9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.868+0000 7f518c9f4640 1 -- 172.21.15.22:0/1610101322 >> v1:172.21.15.22:6790/0 conn(0x7f51881457f0 legacy=0x7f518813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.868+0000 7f518c9f4640 1 -- 172.21.15.22:0/1610101322 shutdown_connections 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.868+0000 7f518c9f4640 1 -- 172.21.15.22:0/1610101322 >> 172.21.15.22:0/1610101322 conn(0x7f5188066a00 msgr2=0x7f5188066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:25.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.868+0000 7f518c9f4640 1 -- 172.21.15.22:0/1610101322 shutdown_connections 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.868+0000 7f518c9f4640 1 -- 172.21.15.22:0/1610101322 wait complete. 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient: build_initial_monmap 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient: monmap: 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 librados: starting msgr at 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 librados: starting objecter 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 Processor -- start 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 -- start start 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 librados: setting wanted keys 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 librados: calling monclient init 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient: init 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient(hunting): picked mon.noname-b con 0x7f51880a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient(hunting): picked mon.noname-c con 0x7f518813f650 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 10 monclient(hunting): picked mon.noname-a con 0x7f51881457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51880b1430 con 0x7f51881457f0 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51880b15c0 con 0x7f51880a9d70 2024-03-29T10:54:25.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.869+0000 7f518c9f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188144840 con 0x7f518813f650 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 10 monclient(hunting): authenticate will time out at 767.174092s 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5185d9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f518813f650 0x7f5188141cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55640/0 (socket says 172.21.15.22:55640) 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5185d9b640 1 -- 172.21.15.22:0/4004628365 learned_addr learned my addr 172.21.15.22:0/4004628365 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 client.?.objecter ms_handle_connect 0x7f518813f650 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 client.?.objecter ms_handle_connect 0x7f51881457f0 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 client.?.objecter ms_handle_connect 0x7f51880a9d70 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235093443 0 0) 0x7f51880b1430 con 0x7f51881457f0 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): my global_id is 4194 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5168004eb0 con 0x7f51881457f0 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179867309 0 0) 0x7f5188144840 con 0x7f518813f650 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): my global_id is 4199 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51880b1430 con 0x7f518813f650 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139566341 0 0) 0x7f5188146340 con 0x7f51880a9d70 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): my global_id is 4198 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5188144840 con 0x7f51880a9d70 2024-03-29T10:54:25.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1620839255 0 0) 0x7f5168004eb0 con 0x7f51881457f0 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5188146340 con 0x7f51881457f0 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1546604019 0 0) 0x7f51880b1430 con 0x7f518813f650 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5168004eb0 con 0x7f518813f650 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5180001bf0 con 0x7f51881457f0 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.866 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1946587059 0 0) 0x7f5188144840 con 0x7f51880a9d70 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51880b1430 con 0x7f51880a9d70 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f517c001290 con 0x7f518813f650 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:25.867 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5178001cb0 con 0x7f51880a9d70 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient(hunting): dump: 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.868 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 89412355 0 0) 0x7f5168004eb0 con 0x7f518813f650 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 >> v1:172.21.15.26:6789/0 conn(0x7f51880a9d70 legacy=0x7f51881d7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 >> v1:172.21.15.22:6789/0 conn(0x7f51881457f0 legacy=0x7f5188142470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 monclient: found mon.c 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:25.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51881449d0 con 0x7f518813f650 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f5176ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 5 monclient: authenticate success, global_id 4199 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 10 monclient: _renew_subs 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51881e3030 con 0x7f518813f650 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 10 monclient: _renew_subs 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f51881e33a0 con 0x7f518813f650 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.870+0000 7f518c9f4640 1 librados: init done 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f517c0013f0 con 0x7f518813f650 2024-03-29T10:54:25.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f517c004a90 con 0x7f518813f650 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 10 monclient: dump: 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:25.871 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5184d99640 4 set_mon_vals no callback set 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f517c004f80 con 0x7f518813f650 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f517c0068f0 con 0x7f518813f650 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 10 client.4199.objecter ms_dispatch 0x7f5188000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 3 client.4199.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 3 client.4199.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f5176ffd640 20 client.4199.objecter dump_active .. 0 homeless 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f518c9f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f518c9f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f518c9f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.871+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f516c005080 con 0x7f518813f650 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.873+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f517c006bd0 con 0x7f518813f650 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.873+0000 7f5176ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:25.872 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.873+0000 7f5176ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:25.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.936+0000 7f518c9f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:25.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.936+0000 7f518c9f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:25.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.936+0000 7f518c9f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:54:25.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.936+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f516c005330 con 0x7f518813f650 2024-03-29T10:54:25.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f5176ffd640 1 -- 172.21.15.22:0/4004628365 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f517c006220 con 0x7f518813f650 2024-03-29T10:54:25.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f5176ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:25.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f5176ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:25.918 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:54:25.918 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:25.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f518c9f4640 10 librados: watch_flush enter 2024-03-29T10:54:25.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f518c9f4640 10 librados: watch_flush exit 2024-03-29T10:54:25.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f518c9f4640 20 client.4199.objecter shutdown clearing up homeless session... 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f518c9f4640 10 client.4199.objecter successfully canceled tick 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f518c9f4640 10 monclient: shutdown 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f518c9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.937+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 >> v1:172.21.15.22:6790/0 conn(0x7f518813f650 legacy=0x7f5188141cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.938+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 shutdown_connections 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.938+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 >> 172.21.15.22:0/4004628365 conn(0x7f5188066a00 msgr2=0x7f51880b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.938+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 shutdown_connections 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.938+0000 7f518c9f4640 1 -- 172.21.15.22:0/4004628365 wait complete. 2024-03-29T10:54:25.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:25.938+0000 7f518c9f4640 1 librados: shutdown 2024-03-29T10:54:25.927 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:26.340 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:26.359+0000 7f82c8e4c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:54:26.356 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:26.375+0000 7f9426830040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T10:54:26.627 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:26.646+0000 7f82c8e4c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:54:26.642 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:26.662+0000 7f9426830040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T10:54:26.681 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:26.700+0000 7f82c8e4c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:54:26.696 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:26.715+0000 7f9426830040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T10:54:26.814 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:26.834+0000 7f82c8e4c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:54:26.815 INFO:tasks.ceph.mgr.x.smithi026.stderr:2024-03-29T10:54:26.835+0000 7f9426830040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T10:54:27.828 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:27.847+0000 7f8269b22640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T10:54:27.828 INFO:tasks.ceph.mgr.y.smithi022.stderr:2024-03-29T10:54:27.848+0000 7f8269b22640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T10:54:28.791 INFO:tasks.ceph.osd.6.smithi026.stderr:2024-03-29T10:54:28.810+0000 7f53af6a6640 -1 osd.6 0 log_to_monitors true 2024-03-29T10:54:28.817 INFO:tasks.ceph.osd.0.smithi022.stderr:2024-03-29T10:54:28.836+0000 7f7a58e14640 -1 osd.0 0 log_to_monitors true 2024-03-29T10:54:28.835 INFO:tasks.ceph.osd.5.smithi026.stderr:2024-03-29T10:54:28.854+0000 7fbbcd40d640 -1 osd.5 0 log_to_monitors true 2024-03-29T10:54:28.853 INFO:tasks.ceph.osd.2.smithi022.stderr:2024-03-29T10:54:28.872+0000 7f744b01a640 -1 osd.2 0 log_to_monitors true 2024-03-29T10:54:28.858 INFO:tasks.ceph.osd.1.smithi022.stderr:2024-03-29T10:54:28.876+0000 7f409820f640 -1 osd.1 0 log_to_monitors true 2024-03-29T10:54:28.883 INFO:tasks.ceph.osd.3.smithi022.stderr:2024-03-29T10:54:28.902+0000 7fed6f8d5640 -1 osd.3 0 log_to_monitors true 2024-03-29T10:54:28.885 INFO:tasks.ceph.osd.7.smithi026.stderr:2024-03-29T10:54:28.904+0000 7f9b2d613640 -1 osd.7 0 log_to_monitors true 2024-03-29T10:54:28.897 INFO:tasks.ceph.osd.4.smithi026.stderr:2024-03-29T10:54:28.915+0000 7fbbc0319640 -1 osd.4 0 log_to_monitors true 2024-03-29T10:54:32.734 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:32.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.903+0000 7fb9509f5640 10 monclient: get_monmap_and_config 2024-03-29T10:54:32.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.903+0000 7fb9509f5640 10 monclient: build_initial_monmap 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.903+0000 7fb9509f5640 10 monclient: monmap: 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:54:32.904464+0000 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:54:32.904464+0000 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:32.885 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:32.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.904+0000 7fb9509f5640 1 Processor -- start 2024-03-29T10:54:32.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.904+0000 7fb9509f5640 1 -- start start 2024-03-29T10:54:32.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.904+0000 7fb9509f5640 10 monclient: init 2024-03-29T10:54:32.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:32.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:32.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9440389f0 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 10 monclient(hunting): picked mon.noname-b con 0x7fb944038dd0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9440324e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb944032930 con 0x7fb9440324e0 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb944033960 con 0x7fb9440389f0 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb944033c40 con 0x7fb944038dd0 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb9509f5640 10 monclient(hunting): authenticate will time out at 774.209924s 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb94b59e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb9440324e0 0x7fb9440cb820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58420/0 (socket says 172.21.15.22:58420) 2024-03-29T10:54:32.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.905+0000 7fb94b59e640 1 -- 172.21.15.22:0/98485325 learned_addr learned my addr 172.21.15.22:0/98485325 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968116139 0 0) 0x7fb944033960 con 0x7fb9440389f0 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient(hunting): my global_id is 4240 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb934003520 con 0x7fb9440389f0 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2965177165 0 0) 0x7fb934003520 con 0x7fb9440389f0 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 >> v1:172.21.15.22:6790/0 conn(0x7fb944038dd0 legacy=0x7fb9440cb110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 >> v1:172.21.15.22:6789/0 conn(0x7fb9440324e0 legacy=0x7fb9440cb820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 1 monclient: found mon.noname-c 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944032b70 con 0x7fb9440389f0 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient: _finish_auth 0 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.906+0000 7fb949d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.907+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb93c001690 con 0x7fb9440389f0 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.907+0000 7fb949d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.907+0000 7fb949d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.907+0000 7fb949d9b640 10 monclient: dump: 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:32.889 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.907+0000 7fb9509f5640 5 monclient: authenticate success, global_id 4240 2024-03-29T10:54:32.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.907+0000 7fb9509f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:32.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb93c001860 con 0x7fb9440389f0 2024-03-29T10:54:32.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb949d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:32.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb949d9b640 1 -- 172.21.15.22:0/98485325 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb93c001c70 con 0x7fb9440389f0 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb949d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb949d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb949d9b640 10 monclient: dump: 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:32.901 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb9509f5640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.908+0000 7fb9509f5640 4 set_mon_vals no callback set 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.909+0000 7fb9509f5640 10 monclient: shutdown 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.909+0000 7fb9509f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.909+0000 7fb9509f5640 1 -- 172.21.15.22:0/98485325 >> v1:172.21.15.26:6789/0 conn(0x7fb9440389f0 legacy=0x7fb9440c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.909+0000 7fb9509f5640 1 -- 172.21.15.22:0/98485325 shutdown_connections 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.909+0000 7fb9509f5640 1 -- 172.21.15.22:0/98485325 >> 172.21.15.22:0/98485325 conn(0x7fb944005cc0 msgr2=0x7fb9440060a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:32.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.909+0000 7fb9509f5640 1 -- 172.21.15.22:0/98485325 shutdown_connections 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.909+0000 7fb9509f5640 1 -- 172.21.15.22:0/98485325 wait complete. 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.910+0000 7fb9509f5640 10 monclient: build_initial_monmap 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.910+0000 7fb9509f5640 10 monclient: monmap: 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:32.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.911+0000 7fb9509f5640 1 librados: starting msgr at 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.911+0000 7fb9509f5640 1 librados: starting objecter 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.911+0000 7fb9509f5640 1 Processor -- start 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.911+0000 7fb9509f5640 1 -- start start 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.911+0000 7fb9509f5640 1 librados: setting wanted keys 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.911+0000 7fb9509f5640 1 librados: calling monclient init 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.911+0000 7fb9509f5640 10 monclient: init 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9509f5640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9509f5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9509f5640 10 monclient(hunting): picked mon.noname-c con 0x7fb9440324e0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb94ad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb9440324e0 0x7fb94402fb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53498/0 (socket says 172.21.15.22:53498) 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb94ad9d640 1 -- 172.21.15.22:0/465782586 learned_addr learned my addr 172.21.15.22:0/465782586 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9509f5640 10 monclient(hunting): picked mon.noname-a con 0x7fb9440389f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9509f5640 10 monclient(hunting): picked mon.noname-b con 0x7fb94402da10 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9337fe640 10 client.?.objecter ms_handle_connect 0x7fb9440324e0 2024-03-29T10:54:32.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94402f410 con 0x7fb9440389f0 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.912+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94402f6b0 con 0x7fb94402da10 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb94402eff0 con 0x7fb9440324e0 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9509f5640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9509f5640 10 monclient(hunting): authenticate will time out at 774.217139s 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb94ad9d640 1 -- 172.21.15.22:0/465782586 >> v1:172.21.15.22:6790/0 conn(0x7fb9440324e0 legacy=0x7fb94402fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb94ad9d640 1 -- 172.21.15.22:0/465782586 >> v1:172.21.15.22:6790/0 conn(0x7fb9440324e0 legacy=0x7fb94402fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb94ad9d640 1 --1- 172.21.15.22:0/465782586 >> v1:172.21.15.22:6790/0 conn(0x7fb9440324e0 0x7fb94402fb90 :-1 s=OPENED pgs=19905 cs=1 l=1).handle_message read tag failed 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb94ad9d640 1 --1- 172.21.15.22:0/465782586 >> v1:172.21.15.22:6790/0 conn(0x7fb9440324e0 0x7fb94402fb90 :-1 s=OPENED pgs=19905 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9337fe640 10 client.?.objecter ms_handle_connect 0x7fb9440389f0 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9337fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.22:6790/0 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9337fe640 10 client.?.objecter ms_handle_connect 0x7fb94402da10 2024-03-29T10:54:32.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.913+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 186013249 0 0) 0x7fb94402f410 con 0x7fb9440389f0 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): my global_id is 4239 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb938004eb0 con 0x7fb9440389f0 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871360533 0 0) 0x7fb94402f6b0 con 0x7fb94402da10 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): my global_id is 4243 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb94402f410 con 0x7fb94402da10 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3161405032 0 0) 0x7fb94402f410 con 0x7fb94402da10 2024-03-29T10:54:32.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb94402f6b0 con 0x7fb94402da10 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb94c068810 con 0x7fb94402da10 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): dump: 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:32.907 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.914+0000 7fb9337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1373354350 0 0) 0x7fb938004eb0 con 0x7fb9440389f0 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb94402f410 con 0x7fb9440389f0 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb940001820 con 0x7fb9440389f0 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 10 monclient(hunting): dump: 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.908 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1668763590 0 0) 0x7fb94402f6b0 con 0x7fb94402da10 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 >> v1:172.21.15.22:6790/0 conn(0x7fb9440324e0 legacy=0x7fb94402fb90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 >> v1:172.21.15.22:6789/0 conn(0x7fb9440389f0 legacy=0x7fb944164330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 monclient: found mon.b 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb944165150 con 0x7fb94402da10 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 10 monclient: _finish_auth 0 2024-03-29T10:54:32.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 5 monclient: authenticate success, global_id 4243 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 10 monclient: _renew_subs 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94402d630 con 0x7fb94402da10 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 10 client.4243.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 10 monclient: _renew_subs 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9440bee50 con 0x7fb94402da10 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.915+0000 7fb9509f5640 1 librados: init done 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.916+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb94c04fdd0 con 0x7fb94402da10 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.916+0000 7fb9337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:32.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.916+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb94c04f730 con 0x7fb94402da10 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.916+0000 7fb9337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.916+0000 7fb9337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.916+0000 7fb9337fe640 10 monclient: dump: 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:32.911 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.916+0000 7fb94959a640 4 set_mon_vals no callback set 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.917+0000 7fb9509f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.917+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fb94c075710 con 0x7fb94402da10 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9509f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9509f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92c005080 con 0x7fb94402da10 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb94ad9d640 10 monclient: get_auth_request con 0x7fb93806dc90 auth_method 0 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fb94c0ff3c0 con 0x7fb94402da10 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9337fe640 10 client.4243.objecter ms_dispatch 0x7fb94c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9337fe640 3 client.4243.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9337fe640 3 client.4243.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.918+0000 7fb9337fe640 20 client.4243.objecter dump_active .. 0 homeless 2024-03-29T10:54:32.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.919+0000 7fb9337fe640 10 client.4243.objecter ms_handle_connect 0x7fb93806dc90 2024-03-29T10:54:32.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.926+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb94c0caca0 con 0x7fb94402da10 2024-03-29T10:54:32.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.926+0000 7fb9337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:32.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:32.926+0000 7fb9337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:33.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.306+0000 7fb9509f5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:33.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.306+0000 7fb9509f5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:33.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.306+0000 7fb9509f5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:33.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.306+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb92c005330 con 0x7fb94402da10 2024-03-29T10:54:33.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.306+0000 7fb9337fe640 1 -- 172.21.15.22:0/465782586 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fb94c0ccbd0 con 0x7fb94402da10 2024-03-29T10:54:33.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.307+0000 7fb9337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:33.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.307+0000 7fb9337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:33.288 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:54:33.288 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:33.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 10 librados: watch_flush enter 2024-03-29T10:54:33.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 10 librados: watch_flush exit 2024-03-29T10:54:33.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 20 client.4243.objecter shutdown clearing up homeless session... 2024-03-29T10:54:33.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 10 client.4243.objecter successfully canceled tick 2024-03-29T10:54:33.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 >> v1:172.21.15.22:6816/380221644 conn(0x7fb93806dc90 legacy=0x7fb938070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:33.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 10 monclient: shutdown 2024-03-29T10:54:33.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:33.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.310+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 >> v1:172.21.15.26:6789/0 conn(0x7fb94402da10 legacy=0x7fb944164a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:33.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.311+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 shutdown_connections 2024-03-29T10:54:33.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.311+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 >> 172.21.15.22:0/465782586 conn(0x7fb944005cc0 msgr2=0x7fb9440bb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:33.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.311+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 shutdown_connections 2024-03-29T10:54:33.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.311+0000 7fb9509f5640 1 -- 172.21.15.22:0/465782586 wait complete. 2024-03-29T10:54:33.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:33.311+0000 7fb9509f5640 1 librados: shutdown 2024-03-29T10:54:33.306 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:40.113 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:40.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.293+0000 7f0ecb50e640 10 monclient: get_monmap_and_config 2024-03-29T10:54:40.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.293+0000 7f0ecb50e640 10 monclient: build_initial_monmap 2024-03-29T10:54:40.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.293+0000 7f0ecb50e640 10 monclient: monmap: 2024-03-29T10:54:40.277 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:54:40.294117+0000 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:54:40.294117+0000 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:40.278 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.294+0000 7f0ecb50e640 1 Processor -- start 2024-03-29T10:54:40.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.294+0000 7f0ecb50e640 1 -- start start 2024-03-29T10:54:40.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.294+0000 7f0ecb50e640 10 monclient: init 2024-03-29T10:54:40.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:40.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:40.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec40b4790 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec40aea90 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec40b39f0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec40b0c60 con 0x7f0ec40b4790 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec40af020 con 0x7f0ec40aea90 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec413fbd0 con 0x7f0ec40b39f0 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ecb50e640 10 monclient(hunting): authenticate will time out at 781.599642s 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ec9aab640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0ec40b39f0 0x7f0ec413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49360/0 (socket says 172.21.15.22:49360) 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.295+0000 7f0ec9aab640 1 -- 172.21.15.22:0/1155556338 learned_addr learned my addr 172.21.15.22:0/1155556338 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858429017 0 0) 0x7f0ec413fbd0 con 0x7f0ec40b39f0 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): my global_id is 4247 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0eb0003520 con 0x7f0ec40b39f0 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28982711 0 0) 0x7f0ec40af020 con 0x7f0ec40aea90 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): my global_id is 4246 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec413fbd0 con 0x7f0ec40aea90 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562492608 0 0) 0x7f0ec40b0c60 con 0x7f0ec40b4790 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 10 monclient(hunting): my global_id is 4242 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.296+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec40af020 con 0x7f0ec40b4790 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1739225730 0 0) 0x7f0eb0003520 con 0x7f0ec40b39f0 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 >> v1:172.21.15.26:6789/0 conn(0x7f0ec40aea90 legacy=0x7f0ec40af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 >> v1:172.21.15.22:6789/0 conn(0x7f0ec40b4790 legacy=0x7f0ec40b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 1 monclient: found mon.noname-b 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec413fd60 con 0x7f0ec40b39f0 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 10 monclient: _finish_auth 0 2024-03-29T10:54:40.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0ecb50e640 5 monclient: authenticate success, global_id 4247 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0ecb50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0ec0001690 con 0x7f0ec40b39f0 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.297+0000 7f0eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 10 monclient: dump: 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:40.291 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0ecb50e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ec00018b0 con 0x7f0ec40b39f0 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 1 -- 172.21.15.22:0/1155556338 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0ec0001c40 con 0x7f0ec40b39f0 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0eb7fff640 10 monclient: dump: 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.292 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0ecb50e640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.298+0000 7f0ecb50e640 4 set_mon_vals no callback set 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.299+0000 7f0ecb50e640 10 monclient: shutdown 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.300+0000 7f0ecb50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.300+0000 7f0ecb50e640 1 -- 172.21.15.22:0/1155556338 >> v1:172.21.15.22:6790/0 conn(0x7f0ec40b39f0 legacy=0x7f0ec413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.300+0000 7f0ecb50e640 1 -- 172.21.15.22:0/1155556338 shutdown_connections 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.300+0000 7f0ecb50e640 1 -- 172.21.15.22:0/1155556338 >> 172.21.15.22:0/1155556338 conn(0x7f0ec4066a00 msgr2=0x7f0ec4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.300+0000 7f0ecb50e640 1 -- 172.21.15.22:0/1155556338 shutdown_connections 2024-03-29T10:54:40.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.300+0000 7f0ecb50e640 1 -- 172.21.15.22:0/1155556338 wait complete. 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.301+0000 7f0ecb50e640 10 monclient: build_initial_monmap 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.301+0000 7f0ecb50e640 10 monclient: monmap: 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.301+0000 7f0ecb50e640 1 librados: starting msgr at 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.301+0000 7f0ecb50e640 1 librados: starting objecter 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.302+0000 7f0ecb50e640 1 Processor -- start 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.302+0000 7f0ecb50e640 1 -- start start 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.302+0000 7f0ecb50e640 1 librados: setting wanted keys 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.302+0000 7f0ecb50e640 1 librados: calling monclient init 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.302+0000 7f0ecb50e640 10 monclient: init 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 10 monclient(hunting): picked mon.noname-a con 0x7f0ec40b39f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 10 monclient(hunting): picked mon.noname-b con 0x7f0ec40b4790 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 10 monclient(hunting): picked mon.noname-c con 0x7f0ec4135e90 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec4136600 con 0x7f0ec40b39f0 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec41362e0 con 0x7f0ec40b4790 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ec4137480 con 0x7f0ec4135e90 2024-03-29T10:54:40.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ecb50e640 10 monclient(hunting): authenticate will time out at 781.607456s 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ec92aa640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f0ec40b39f0 0x7f0ec4133ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:56416/0 (socket says 172.21.15.22:56416) 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0ec92aa640 1 -- 172.21.15.22:0/3243834630 learned_addr learned my addr 172.21.15.22:0/3243834630 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0eb5ffb640 10 client.?.objecter ms_handle_connect 0x7f0ec4135e90 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.303+0000 7f0eb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 client.?.objecter ms_handle_connect 0x7f0ec40b4790 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 client.?.objecter ms_handle_connect 0x7f0ec40b39f0 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568216314 0 0) 0x7f0ec41362e0 con 0x7f0ec40b4790 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): my global_id is 4249 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea8004eb0 con 0x7f0ec40b4790 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144967452 0 0) 0x7f0ec4137480 con 0x7f0ec4135e90 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): my global_id is 4250 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec41362e0 con 0x7f0ec4135e90 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814775948 0 0) 0x7f0ec4136600 con 0x7f0ec40b39f0 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): my global_id is 4245 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ec4137480 con 0x7f0ec40b39f0 2024-03-29T10:54:40.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3413663709 0 0) 0x7f0ea8004eb0 con 0x7f0ec40b4790 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ec4136600 con 0x7f0ec40b4790 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0eb8001970 con 0x7f0ec40b4790 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:40.298 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.304+0000 7f0eb5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2556068075 0 0) 0x7f0ec4137480 con 0x7f0ec40b39f0 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ea8004eb0 con 0x7f0ec40b39f0 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0ebc001c90 con 0x7f0ec40b39f0 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.299 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932651146 0 0) 0x7f0ec41362e0 con 0x7f0ec4135e90 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ec4137480 con 0x7f0ec4135e90 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0ec00015d0 con 0x7f0ec4135e90 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.305+0000 7f0eb5ffb640 10 monclient(hunting): dump: 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.300 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3899534559 0 0) 0x7f0ec4136600 con 0x7f0ec40b4790 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 >> v1:172.21.15.22:6790/0 conn(0x7f0ec4135e90 legacy=0x7f0ec41e23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 >> v1:172.21.15.22:6789/0 conn(0x7f0ec40b39f0 legacy=0x7f0ec4133ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 1 monclient: found mon.b 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ec4137760 con 0x7f0ec40b4790 2024-03-29T10:54:40.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 5 monclient: authenticate success, global_id 4249 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 10 monclient: _renew_subs 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ec40ae1e0 con 0x7f0ec40b4790 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 10 monclient: _renew_subs 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ec41e3b10 con 0x7f0ec40b4790 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0ecb50e640 1 librados: init done 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0eb80035f0 con 0x7f0ec40b4790 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:40.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0eb8003900 con 0x7f0ec40b4790 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb5ffb640 10 monclient: dump: 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:40.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.306+0000 7f0eb77fe640 4 set_mon_vals no callback set 2024-03-29T10:54:40.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.307+0000 7f0ecb50e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.308+0000 7f0ecb50e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.308+0000 7f0ecb50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.308+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ea4005080 con 0x7f0ec40b4790 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.309+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f0eb80198d0 con 0x7f0ec40b4790 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.309+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0eb8083850 con 0x7f0ec40b4790 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.309+0000 7f0eb5ffb640 10 client.4249.objecter ms_dispatch 0x7f0ec4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.309+0000 7f0eb5ffb640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.309+0000 7f0eb5ffb640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.309+0000 7f0eb5ffb640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.309+0000 7f0ec92aa640 10 monclient: get_auth_request con 0x7f0ea806e170 auth_method 0 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.310+0000 7f0eb5ffb640 10 client.4249.objecter ms_handle_connect 0x7f0ea806e170 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.319+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0eb804ef60 con 0x7f0ec40b4790 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.319+0000 7f0eb5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:40.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.319+0000 7f0eb5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:40.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.697+0000 7f0ecb50e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.697+0000 7f0ecb50e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.697+0000 7f0ecb50e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:40.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.697+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ea4005330 con 0x7f0ec40b4790 2024-03-29T10:54:40.679 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:54:40.679 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:40.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.698+0000 7f0eb5ffb640 1 -- 172.21.15.22:0/3243834630 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0eb8051060 con 0x7f0ec40b4790 2024-03-29T10:54:40.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.698+0000 7f0eb5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:40.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.698+0000 7f0eb5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 10 librados: watch_flush enter 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 10 librados: watch_flush exit 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 10 client.4249.objecter successfully canceled tick 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 >> v1:172.21.15.22:6816/380221644 conn(0x7f0ea806e170 legacy=0x7f0ea8070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 10 monclient: shutdown 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 >> v1:172.21.15.26:6789/0 conn(0x7f0ec40b4790 legacy=0x7f0ec41343b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:40.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 shutdown_connections 2024-03-29T10:54:40.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.702+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 >> 172.21.15.22:0/3243834630 conn(0x7f0ec4066a00 msgr2=0x7f0ec40b1080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:40.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.703+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 shutdown_connections 2024-03-29T10:54:40.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.703+0000 7f0ecb50e640 1 -- 172.21.15.22:0/3243834630 wait complete. 2024-03-29T10:54:40.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:40.703+0000 7f0ecb50e640 1 librados: shutdown 2024-03-29T10:54:40.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:47.505 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:47.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.676+0000 7f9ef9864640 10 monclient: get_monmap_and_config 2024-03-29T10:54:47.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.676+0000 7f9ef9864640 10 monclient: build_initial_monmap 2024-03-29T10:54:47.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.676+0000 7f9ef9864640 10 monclient: monmap: 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:54:47.677517+0000 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:54:47.677517+0000 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:47.658 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:47.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.677+0000 7f9ef9864640 1 Processor -- start 2024-03-29T10:54:47.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 1 -- start start 2024-03-29T10:54:47.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient: init 2024-03-29T10:54:47.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:47.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:47.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient(hunting): picked mon.noname-b con 0x7f9ef40743e0 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:47.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient(hunting): picked mon.noname-c con 0x7f9ef40747c0 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient(hunting): picked mon.noname-a con 0x7f9ef413bd30 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef406f120 con 0x7f9ef413bd30 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef4070f40 con 0x7f9ef40747c0 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef4071360 con 0x7f9ef40743e0 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.678+0000 7f9ef9864640 10 monclient(hunting): authenticate will time out at 788.983058s 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.679+0000 7f9ef27fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9ef40747c0 0x7f9ef413b620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:41904/0 (socket says 172.21.15.22:41904) 2024-03-29T10:54:47.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.679+0000 7f9ef27fc640 1 -- 172.21.15.22:0/2598014423 learned_addr learned my addr 172.21.15.22:0/2598014423 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:47.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.681+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626588799 0 0) 0x7f9ef4070f40 con 0x7f9ef40747c0 2024-03-29T10:54:47.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.681+0000 7f9ef1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.681+0000 7f9ef1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.681+0000 7f9ef1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.681+0000 7f9ef1ffb640 10 monclient(hunting): my global_id is 4252 2024-03-29T10:54:47.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.681+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee0003520 con 0x7f9ef40747c0 2024-03-29T10:54:47.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305045174 0 0) 0x7f9ef406f120 con 0x7f9ef413bd30 2024-03-29T10:54:47.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient(hunting): my global_id is 4248 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ef4070f40 con 0x7f9ef413bd30 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2818212682 0 0) 0x7f9ee0003520 con 0x7f9ef40747c0 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 >> v1:172.21.15.22:6790/0 conn(0x7f9ef40743e0 legacy=0x7f9ef406ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 >> v1:172.21.15.22:6789/0 conn(0x7f9ef413bd30 legacy=0x7f9ef413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 monclient: found mon.noname-c 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4072e60 con 0x7f9ef40747c0 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:54:47.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef9864640 5 monclient: authenticate success, global_id 4252 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef9864640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9ee4001a10 con 0x7f9ef40747c0 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef1ffb640 10 monclient: dump: 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:47.667 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.682+0000 7f9ef9864640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.684+0000 7f9ef1ffb640 1 -- 172.21.15.22:0/2598014423 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ee4001c30 con 0x7f9ef40747c0 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.684+0000 7f9ef1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.684+0000 7f9ef9864640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.684+0000 7f9ef9864640 4 set_mon_vals no callback set 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.685+0000 7f9ef9864640 10 monclient: shutdown 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.685+0000 7f9ef9864640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.685+0000 7f9ef9864640 1 -- 172.21.15.22:0/2598014423 >> v1:172.21.15.26:6789/0 conn(0x7f9ef40747c0 legacy=0x7f9ef413b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.685+0000 7f9ef9864640 1 -- 172.21.15.22:0/2598014423 shutdown_connections 2024-03-29T10:54:47.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.685+0000 7f9ef9864640 1 -- 172.21.15.22:0/2598014423 >> 172.21.15.22:0/2598014423 conn(0x7f9ef40b3f40 msgr2=0x7f9ef40b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:47.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.686+0000 7f9ef9864640 1 -- 172.21.15.22:0/2598014423 shutdown_connections 2024-03-29T10:54:47.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.686+0000 7f9ef9864640 1 -- 172.21.15.22:0/2598014423 wait complete. 2024-03-29T10:54:47.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.687+0000 7f9ef9864640 10 monclient: build_initial_monmap 2024-03-29T10:54:47.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.687+0000 7f9ef9864640 10 monclient: monmap: 2024-03-29T10:54:47.669 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:47.669 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:47.670 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:47.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.687+0000 7f9ef9864640 1 librados: starting msgr at 2024-03-29T10:54:47.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.687+0000 7f9ef9864640 1 librados: starting objecter 2024-03-29T10:54:47.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.687+0000 7f9ef9864640 1 Processor -- start 2024-03-29T10:54:47.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.687+0000 7f9ef9864640 1 -- start start 2024-03-29T10:54:47.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 1 librados: setting wanted keys 2024-03-29T10:54:47.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 1 librados: calling monclient init 2024-03-29T10:54:47.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient: init 2024-03-29T10:54:47.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:47.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:54:47.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient(hunting): picked mon.noname-b con 0x7f9ef40743e0 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient(hunting): picked mon.noname-c con 0x7f9ef413bd30 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient(hunting): picked mon.noname-a con 0x7f9ef413ad40 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef41c49b0 con 0x7f9ef413ad40 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef41c4e40 con 0x7f9ef40743e0 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef41c5cc0 con 0x7f9ef413bd30 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.688+0000 7f9ef9864640 10 monclient(hunting): authenticate will time out at 788.992851s 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.691+0000 7f9ef2ffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f9ef40743e0 0x7f9ef41db950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:42078/0 (socket says 172.21.15.22:42078) 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.691+0000 7f9ef2ffd640 1 -- 172.21.15.22:0/4013692234 learned_addr learned my addr 172.21.15.22:0/4013692234 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.691+0000 7f9eebfff640 10 client.?.objecter ms_handle_connect 0x7f9ef40743e0 2024-03-29T10:54:47.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.691+0000 7f9eebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:47.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4153523590 0 0) 0x7f9ef41c4e40 con 0x7f9ef40743e0 2024-03-29T10:54:47.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): my global_id is 4255 2024-03-29T10:54:47.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed4004eb0 con 0x7f9ef40743e0 2024-03-29T10:54:47.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 client.?.objecter ms_handle_connect 0x7f9ef413ad40 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074218762 0 0) 0x7f9ef41c49b0 con 0x7f9ef413ad40 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): _negotiate 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): my global_id is 4251 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ed40051d0 con 0x7f9ef413ad40 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2757557190 0 0) 0x7f9ed4004eb0 con 0x7f9ef40743e0 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ed4005640 con 0x7f9ef40743e0 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 877340749 0 0) 0x7f9ed40051d0 con 0x7f9ef413ad40 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ed4005c40 con 0x7f9ef413ad40 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9eec001810 con 0x7f9ef413ad40 2024-03-29T10:54:47.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): dump: 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:47.676 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9edc001920 con 0x7f9ef40743e0 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.692+0000 7f9eebfff640 10 monclient(hunting): dump: 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:47.677 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:47.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.693+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3425165090 0 0) 0x7f9ed4005c40 con 0x7f9ef413ad40 2024-03-29T10:54:47.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.693+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 >> v1:172.21.15.22:6790/0 conn(0x7f9ef413bd30 legacy=0x7f9ef41367a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:54:47.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 >> v1:172.21.15.26:6789/0 conn(0x7f9ef40743e0 legacy=0x7f9ef41db950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:47.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 10 monclient: _finish_hunting 0 2024-03-29T10:54:47.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 1 monclient: found mon.a 2024-03-29T10:54:47.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:47.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef41c5fa0 con 0x7f9ef413ad40 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 10 monclient: _finish_auth 0 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 10 monclient: _check_auth_tickets 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.694+0000 7f9eebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 5 monclient: authenticate success, global_id 4251 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 10 monclient: _renew_subs 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef41c4580 con 0x7f9ef413ad40 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 10 monclient: _renew_subs 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef413d3b0 con 0x7f9ef413ad40 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef9864640 1 librados: init done 2024-03-29T10:54:47.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9eec001d70 con 0x7f9ef413ad40 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9eebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9eec001810 con 0x7f9ef413ad40 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9eebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9eebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9eebfff640 10 monclient: dump: 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:47.681 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.695+0000 7f9ef0ff9640 4 set_mon_vals no callback set 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.696+0000 7f9ef9864640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.696+0000 7f9ef9864640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.696+0000 7f9ef9864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.696+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed8005080 con 0x7f9ef413ad40 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.697+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f9eec0194e0 con 0x7f9ef413ad40 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.698+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f9eec083400 con 0x7f9ef413ad40 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.698+0000 7f9eebfff640 10 client.4251.objecter ms_dispatch 0x7f9ef4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.698+0000 7f9eebfff640 3 client.4251.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.698+0000 7f9eebfff640 3 client.4251.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.698+0000 7f9eebfff640 20 client.4251.objecter dump_active .. 0 homeless 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.698+0000 7f9ef2ffd640 10 monclient: get_auth_request con 0x7f9ed406e850 auth_method 0 2024-03-29T10:54:47.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.698+0000 7f9eebfff640 10 client.4251.objecter ms_handle_connect 0x7f9ed406e850 2024-03-29T10:54:47.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.706+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9eec04eb10 con 0x7f9ef413ad40 2024-03-29T10:54:47.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.706+0000 7f9eebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:47.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:47.706+0000 7f9eebfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:48.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.050+0000 7f9ef9864640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:48.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.050+0000 7f9ef9864640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:48.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.050+0000 7f9ef9864640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:54:48.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.050+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ed8005330 con 0x7f9ef413ad40 2024-03-29T10:54:48.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.052+0000 7f9eebfff640 1 -- 172.21.15.22:0/4013692234 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f9eec050c10 con 0x7f9ef413ad40 2024-03-29T10:54:48.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.052+0000 7f9eebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:48.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.052+0000 7f9eebfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:48.034 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:54:48.034 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:48.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.056+0000 7f9ef9864640 10 librados: watch_flush enter 2024-03-29T10:54:48.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.056+0000 7f9ef9864640 10 librados: watch_flush exit 2024-03-29T10:54:48.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.056+0000 7f9ef9864640 20 client.4251.objecter shutdown clearing up homeless session... 2024-03-29T10:54:48.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 10 client.4251.objecter successfully canceled tick 2024-03-29T10:54:48.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 >> v1:172.21.15.22:6816/380221644 conn(0x7f9ed406e850 legacy=0x7f9ed4070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:48.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 10 monclient: shutdown 2024-03-29T10:54:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 >> v1:172.21.15.22:6789/0 conn(0x7f9ef413ad40 legacy=0x7f9ef41e7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 shutdown_connections 2024-03-29T10:54:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 >> 172.21.15.22:0/4013692234 conn(0x7f9ef40b3f40 msgr2=0x7f9ef406ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 shutdown_connections 2024-03-29T10:54:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 1 -- 172.21.15.22:0/4013692234 wait complete. 2024-03-29T10:54:48.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:48.057+0000 7f9ef9864640 1 librados: shutdown 2024-03-29T10:54:48.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:54:54.867 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:54:55.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.028+0000 7f6a10e9a640 10 monclient: get_monmap_and_config 2024-03-29T10:54:55.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.028+0000 7f6a10e9a640 10 monclient: build_initial_monmap 2024-03-29T10:54:55.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.028+0000 7f6a10e9a640 10 monclient: monmap: 2024-03-29T10:54:55.011 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:55.011 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:55.011 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:54:55.030165+0000 2024-03-29T10:54:55.011 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:54:55.030165+0000 2024-03-29T10:54:55.012 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:55.012 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:55.012 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:55.012 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:54:55.012 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:54:55.012 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:55.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.030+0000 7f6a10e9a640 1 Processor -- start 2024-03-29T10:54:55.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.030+0000 7f6a10e9a640 1 -- start start 2024-03-29T10:54:55.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.030+0000 7f6a10e9a640 10 monclient: init 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.031+0000 7f6a10e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.031+0000 7f6a10e9a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.031+0000 7f6a10e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0c0ae250 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.031+0000 7f6a10e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0c13f970 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.032+0000 7f6a10e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0c142800 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.032+0000 7f6a10e9a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0afe00 con 0x7f6a0c0ae250 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.032+0000 7f6a10e9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c142c10 con 0x7f6a0c142800 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.032+0000 7f6a10e9a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0afb20 con 0x7f6a0c13f970 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.032+0000 7f6a10e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:55.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.032+0000 7f6a10e9a640 10 monclient(hunting): authenticate will time out at 796.337136s 2024-03-29T10:54:55.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.035+0000 7f6a09d9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6a0c13f970 0x7f6a0c1420f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42729/0 (socket says 172.21.15.22:42729) 2024-03-29T10:54:55.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.035+0000 7f6a09d9b640 1 -- 172.21.15.22:0/2812415486 learned_addr learned my addr 172.21.15.22:0/2812415486 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:55.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.036+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230374265 0 0) 0x7f6a0c0afb20 con 0x7f6a0c13f970 2024-03-29T10:54:55.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.036+0000 7f6a0959a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.036+0000 7f6a0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.036+0000 7f6a0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.036+0000 7f6a0959a640 10 monclient(hunting): my global_id is 4253 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.036+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f4003520 con 0x7f6a0c13f970 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190922497 0 0) 0x7f6a0c0afe00 con 0x7f6a0c0ae250 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient(hunting): my global_id is 4254 2024-03-29T10:54:55.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0c0afb20 con 0x7f6a0c0ae250 2024-03-29T10:54:55.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3705332754 0 0) 0x7f69f4003520 con 0x7f6a0c13f970 2024-03-29T10:54:55.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 >> v1:172.21.15.26:6789/0 conn(0x7f6a0c142800 legacy=0x7f6a0c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:54:55.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 >> v1:172.21.15.22:6789/0 conn(0x7f6a0c0ae250 legacy=0x7f6a0c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: _finish_hunting 0 2024-03-29T10:54:55.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 monclient: found mon.noname-b 2024-03-29T10:54:55.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c0aab80 con 0x7f6a0c13f970 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: _finish_auth 0 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: _check_auth_tickets 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f69fc001690 con 0x7f6a0c13f970 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:55.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: dump: 2024-03-29T10:54:55.022 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:55.022 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:55.022 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.022 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.022 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.026 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:55.026 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:55.026 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:55.026 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:55.026 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69fc0018b0 con 0x7f6a0c13f970 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 1 -- 172.21.15.22:0/2812415486 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f69fc001c40 con 0x7f6a0c13f970 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.037+0000 7f6a0959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.038+0000 7f6a0959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.038+0000 7f6a0959a640 10 monclient: dump: 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.027 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:55.028 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:55.028 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:55.028 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:55.028 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:55.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.038+0000 7f6a10e9a640 5 monclient: authenticate success, global_id 4253 2024-03-29T10:54:55.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.038+0000 7f6a10e9a640 10 monclient: get_monmap_and_config success 2024-03-29T10:54:55.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.038+0000 7f6a10e9a640 4 set_mon_vals no callback set 2024-03-29T10:54:55.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.039+0000 7f6a10e9a640 10 monclient: shutdown 2024-03-29T10:54:55.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.039+0000 7f6a10e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:55.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.039+0000 7f6a10e9a640 1 -- 172.21.15.22:0/2812415486 >> v1:172.21.15.22:6790/0 conn(0x7f6a0c13f970 legacy=0x7f6a0c1420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.040+0000 7f6a10e9a640 1 -- 172.21.15.22:0/2812415486 shutdown_connections 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.040+0000 7f6a10e9a640 1 -- 172.21.15.22:0/2812415486 >> 172.21.15.22:0/2812415486 conn(0x7f6a0c0b35e0 msgr2=0x7f6a0c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.040+0000 7f6a10e9a640 1 -- 172.21.15.22:0/2812415486 shutdown_connections 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.040+0000 7f6a10e9a640 1 -- 172.21.15.22:0/2812415486 wait complete. 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.041+0000 7f6a10e9a640 10 monclient: build_initial_monmap 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.041+0000 7f6a10e9a640 10 monclient: monmap: 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:54:55.037 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.042+0000 7f6a10e9a640 1 librados: starting msgr at 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.042+0000 7f6a10e9a640 1 librados: starting objecter 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.042+0000 7f6a10e9a640 1 Processor -- start 2024-03-29T10:54:55.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.042+0000 7f6a10e9a640 1 -- start start 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.042+0000 7f6a10e9a640 1 librados: setting wanted keys 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.042+0000 7f6a10e9a640 1 librados: calling monclient init 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.042+0000 7f6a10e9a640 10 monclient: init 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.043+0000 7f6a10e9a640 10 monclient: _reopen_session rank -1 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.043+0000 7f6a10e9a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.043+0000 7f6a10e9a640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0c0ae250 addr v1:172.21.15.22:6790/0 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.043+0000 7f6a10e9a640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0c13f970 addr v1:172.21.15.26:6789/0 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.044+0000 7f6a10e9a640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0c142800 addr v1:172.21.15.22:6789/0 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.044+0000 7f6a10e9a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c145d90 con 0x7f6a0c142800 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.044+0000 7f6a10e9a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c1387f0 con 0x7f6a0c13f970 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.044+0000 7f6a10e9a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c144ce0 con 0x7f6a0c0ae250 2024-03-29T10:54:55.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.044+0000 7f6a10e9a640 10 monclient(hunting): _renew_subs 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.044+0000 7f6a10e9a640 10 monclient(hunting): authenticate will time out at 796.349984s 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.045+0000 7f6a09d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6a0c13f970 0x7f6a0c071a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:35402/0 (socket says 172.21.15.22:35402) 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.045+0000 7f6a09d9b640 1 -- 172.21.15.22:0/335208684 learned_addr learned my addr 172.21.15.22:0/335208684 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 10 client.?.objecter ms_handle_connect 0x7f6a0c13f970 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1373393691 0 0) 0x7f6a0c1387f0 con 0x7f6a0c13f970 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 10 monclient(hunting): my global_id is 4258 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.046+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec004eb0 con 0x7f6a0c13f970 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1818501338 0 0) 0x7f69ec004eb0 con 0x7f6a0c13f970 2024-03-29T10:54:55.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a0c1387f0 con 0x7f6a0c13f970 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f69fc0016d0 con 0x7f6a0c13f970 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): dump: 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:55.041 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:55.042 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:55.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:54:55.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 client.?.objecter ms_handle_connect 0x7f6a0c142800 2024-03-29T10:54:55.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:55.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 client.?.objecter ms_handle_connect 0x7f6a0c0ae250 2024-03-29T10:54:55.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:54:55.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879189676 0 0) 0x7f6a0c145d90 con 0x7f6a0c142800 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): my global_id is 4257 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69ec004eb0 con 0x7f6a0c142800 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977937262 0 0) 0x7f6a0c144ce0 con 0x7f6a0c0ae250 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): _negotiate 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 10 monclient(hunting): my global_id is 4256 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.047+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a0c145d90 con 0x7f6a0c0ae250 2024-03-29T10:54:55.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4285246495 0 0) 0x7f6a0c1387f0 con 0x7f6a0c13f970 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 >> v1:172.21.15.22:6790/0 conn(0x7f6a0c0ae250 legacy=0x7f6a0c144410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 >> v1:172.21.15.22:6789/0 conn(0x7f6a0c142800 legacy=0x7f6a0c1da560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 10 monclient: _finish_hunting 0 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 1 monclient: found mon.b 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a0c143ef0 con 0x7f6a0c13f970 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 10 monclient: _finish_auth 0 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 10 monclient: _check_auth_tickets 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f69faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 5 monclient: authenticate success, global_id 4258 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 10 monclient: _renew_subs 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:55.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a0c072110 con 0x7f6a0c13f970 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 10 monclient: _renew_subs 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0c072640 con 0x7f6a0c13f970 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.048+0000 7f6a10e9a640 1 librados: init done 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69fc004a90 con 0x7f6a0c13f970 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f69faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f69fc004ea0 con 0x7f6a0c13f970 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f69faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f69faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f69faffd640 10 monclient: dump: 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:54:55.045 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f69fbfff640 4 set_mon_vals no callback set 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.049+0000 7f6a10e9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.050+0000 7f6a10e9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.050+0000 7f6a10e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.050+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f0005080 con 0x7f6a0c13f970 2024-03-29T10:54:55.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.051+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f69fc005000 con 0x7f6a0c13f970 2024-03-29T10:54:55.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.052+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f69fc084cc0 con 0x7f6a0c13f970 2024-03-29T10:54:55.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.052+0000 7f69faffd640 10 client.4258.objecter ms_dispatch 0x7f6a0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:54:55.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.052+0000 7f69faffd640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:54:55.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.052+0000 7f69faffd640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:54:55.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.052+0000 7f69faffd640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-29T10:54:55.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.052+0000 7f6a0a59c640 10 monclient: get_auth_request con 0x7f69ec06de30 auth_method 0 2024-03-29T10:54:55.047 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.053+0000 7f69faffd640 10 client.4258.objecter ms_handle_connect 0x7f69ec06de30 2024-03-29T10:54:55.048 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.059+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69fc0505a0 con 0x7f6a0c13f970 2024-03-29T10:54:55.048 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.059+0000 7f69faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:54:55.048 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.059+0000 7f69faffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:54:55.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.384+0000 7f6a10e9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:55.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.384+0000 7f6a10e9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:55.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.384+0000 7f6a10e9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:54:55.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.384+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69f0005330 con 0x7f6a0c13f970 2024-03-29T10:54:55.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.387+0000 7f69faffd640 1 -- 172.21.15.22:0/335208684 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f69fc0524d0 con 0x7f6a0c13f970 2024-03-29T10:54:55.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.387+0000 7f69faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:54:55.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.387+0000 7f69faffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:54:55.370 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:54:55.370 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:54:55.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.390+0000 7f6a10e9a640 10 librados: watch_flush enter 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.390+0000 7f6a10e9a640 10 librados: watch_flush exit 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.390+0000 7f6a10e9a640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.391+0000 7f6a10e9a640 10 client.4258.objecter successfully canceled tick 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.391+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 >> v1:172.21.15.22:6816/380221644 conn(0x7f69ec06de30 legacy=0x7f69ec0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.391+0000 7f6a10e9a640 10 monclient: shutdown 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.391+0000 7f6a10e9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.391+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 >> v1:172.21.15.26:6789/0 conn(0x7f6a0c13f970 legacy=0x7f6a0c071a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.391+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 shutdown_connections 2024-03-29T10:54:55.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.391+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 >> 172.21.15.22:0/335208684 conn(0x7f6a0c0b35e0 msgr2=0x7f6a0c13e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:54:55.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.392+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 shutdown_connections 2024-03-29T10:54:55.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.393+0000 7f6a10e9a640 1 -- 172.21.15.22:0/335208684 wait complete. 2024-03-29T10:54:55.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:54:55.393+0000 7f6a10e9a640 1 librados: shutdown 2024-03-29T10:54:55.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:02.201 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:02.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.373+0000 7ff79ca7b640 10 monclient: get_monmap_and_config 2024-03-29T10:55:02.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.373+0000 7ff79ca7b640 10 monclient: build_initial_monmap 2024-03-29T10:55:02.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.374+0000 7ff79ca7b640 10 monclient: monmap: 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:02.375805+0000 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:02.375805+0000 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:02.357 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:02.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.378+0000 7ff79ca7b640 1 Processor -- start 2024-03-29T10:55:02.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.378+0000 7ff79ca7b640 1 -- start start 2024-03-29T10:55:02.360 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.378+0000 7ff79ca7b640 10 monclient: init 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 10 monclient(hunting): picked mon.noname-b con 0x7ff7980b46b0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7980b3da0 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 10 monclient(hunting): picked mon.noname-c con 0x7ff798142a40 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980b3530 con 0x7ff7980b3da0 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980b4280 con 0x7ff798142a40 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7980aa9f0 con 0x7ff7980b46b0 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:02.361 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.379+0000 7ff79ca7b640 10 monclient(hunting): authenticate will time out at 803.684484s 2024-03-29T10:55:02.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.385+0000 7ff79659c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff7980b46b0 0x7ff798142330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39011/0 (socket says 172.21.15.22:39011) 2024-03-29T10:55:02.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.385+0000 7ff79659c640 1 -- 172.21.15.22:0/3295132028 learned_addr learned my addr 172.21.15.22:0/3295132028 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:02.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285153301 0 0) 0x7ff7980aa9f0 con 0x7ff7980b46b0 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): my global_id is 4259 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff780003520 con 0x7ff7980b46b0 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657022268 0 0) 0x7ff78c001110 con 0x7ff798142a40 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 10 monclient(hunting): my global_id is 4261 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7980aa9f0 con 0x7ff798142a40 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.386+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624953140 0 0) 0x7ff7980b3530 con 0x7ff7980b3da0 2024-03-29T10:55:02.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient(hunting): my global_id is 4260 2024-03-29T10:55:02.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff78c001110 con 0x7ff7980b3da0 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 209491112 0 0) 0x7ff7980aa9f0 con 0x7ff798142a40 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 >> v1:172.21.15.22:6790/0 conn(0x7ff7980b46b0 legacy=0x7ff798142330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 >> v1:172.21.15.22:6789/0 conn(0x7ff7980b3da0 legacy=0x7ff79813b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 1 monclient: found mon.noname-c 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:55:02.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7980aacd0 con 0x7ff798142a40 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient: _finish_auth 0 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff78c0015e0 con 0x7ff798142a40 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79559a640 10 monclient: dump: 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:02.372 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:02.373 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:02.373 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:02.373 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:02.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79ca7b640 5 monclient: authenticate success, global_id 4261 2024-03-29T10:55:02.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.387+0000 7ff79ca7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.390+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff78c001740 con 0x7ff798142a40 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.390+0000 7ff79559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.390+0000 7ff79ca7b640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.390+0000 7ff79ca7b640 4 set_mon_vals no callback set 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.391+0000 7ff79559a640 1 -- 172.21.15.22:0/3295132028 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff78c001b90 con 0x7ff798142a40 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.391+0000 7ff79559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.391+0000 7ff79559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.391+0000 7ff79559a640 10 monclient: dump: 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:02.374 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.392+0000 7ff79ca7b640 10 monclient: shutdown 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.392+0000 7ff79ca7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.392+0000 7ff79ca7b640 1 -- 172.21.15.22:0/3295132028 >> v1:172.21.15.26:6789/0 conn(0x7ff798142a40 legacy=0x7ff79814d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.392+0000 7ff79ca7b640 1 -- 172.21.15.22:0/3295132028 shutdown_connections 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.392+0000 7ff79ca7b640 1 -- 172.21.15.22:0/3295132028 >> 172.21.15.22:0/3295132028 conn(0x7ff79806d0f0 msgr2=0x7ff798066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:02.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.392+0000 7ff79ca7b640 1 -- 172.21.15.22:0/3295132028 shutdown_connections 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.392+0000 7ff79ca7b640 1 -- 172.21.15.22:0/3295132028 wait complete. 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.393+0000 7ff79ca7b640 10 monclient: build_initial_monmap 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.393+0000 7ff79ca7b640 10 monclient: monmap: 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:02.376 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.394+0000 7ff79ca7b640 1 librados: starting msgr at 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.394+0000 7ff79ca7b640 1 librados: starting objecter 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.394+0000 7ff79ca7b640 1 Processor -- start 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.394+0000 7ff79ca7b640 1 -- start start 2024-03-29T10:55:02.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.394+0000 7ff79ca7b640 1 librados: setting wanted keys 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.394+0000 7ff79ca7b640 1 librados: calling monclient init 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.394+0000 7ff79ca7b640 10 monclient: init 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 10 monclient(hunting): picked mon.noname-a con 0x7ff7980b3da0 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 10 monclient(hunting): picked mon.noname-c con 0x7ff7980b46b0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 10 monclient(hunting): picked mon.noname-b con 0x7ff798142a40 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79813d680 con 0x7ff7980b3da0 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff798134520 con 0x7ff798142a40 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff79813d8b0 con 0x7ff7980b46b0 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:02.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.395+0000 7ff79ca7b640 10 monclient(hunting): authenticate will time out at 803.700432s 2024-03-29T10:55:02.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.399+0000 7ff795d9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff7980b46b0 0x7ff7981384b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39089/0 (socket says 172.21.15.22:39089) 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.399+0000 7ff795d9b640 1 -- 172.21.15.22:0/913318967 learned_addr learned my addr 172.21.15.22:0/913318967 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 client.?.objecter ms_handle_connect 0x7ff7980b46b0 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3721060860 0 0) 0x7ff79813d8b0 con 0x7ff7980b46b0 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): my global_id is 4262 2024-03-29T10:55:02.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff778004eb0 con 0x7ff7980b46b0 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 673043766 0 0) 0x7ff778004eb0 con 0x7ff7980b46b0 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff79813d8b0 con 0x7ff7980b46b0 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff7880019b0 con 0x7ff7980b46b0 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.383 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.384 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:02.384 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:02.384 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:02.384 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:02.384 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:02.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.400+0000 7ff786ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:02.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.403+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 925696395 0 0) 0x7ff79813d8b0 con 0x7ff7980b46b0 2024-03-29T10:55:02.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.403+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 >> v1:172.21.15.26:6789/0 conn(0x7ff798142a40 legacy=0x7ff7981e5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.403+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 >> v1:172.21.15.22:6789/0 conn(0x7ff7980b3da0 legacy=0x7ff798137da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.403+0000 7ff786ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 1 monclient: found mon.c 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79813a880 con 0x7ff7980b46b0 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 5 monclient: authenticate success, global_id 4262 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 10 monclient: _renew_subs 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79813cc30 con 0x7ff7980b46b0 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 10 client.4262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 10 monclient: _renew_subs 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:02.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79813c880 con 0x7ff7980b46b0 2024-03-29T10:55:02.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff79ca7b640 1 librados: init done 2024-03-29T10:55:02.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff788001c10 con 0x7ff7980b46b0 2024-03-29T10:55:02.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff786ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:02.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.404+0000 7ff794d99640 4 set_mon_vals no callback set 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.405+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff788003a60 con 0x7ff7980b46b0 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.405+0000 7ff786ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.405+0000 7ff786ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.405+0000 7ff786ffd640 10 monclient: dump: 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:02.389 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.406+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7ff788003bc0 con 0x7ff7980b46b0 2024-03-29T10:55:02.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.407+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7ff788084c50 con 0x7ff7980b46b0 2024-03-29T10:55:02.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.407+0000 7ff786ffd640 10 client.4262.objecter ms_dispatch 0x7ff798000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:02.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.407+0000 7ff786ffd640 3 client.4262.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.407+0000 7ff786ffd640 3 client.4262.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.407+0000 7ff786ffd640 20 client.4262.objecter dump_active .. 0 homeless 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.408+0000 7ff79ca7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.408+0000 7ff79ca7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.408+0000 7ff79ca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.408+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff77c005080 con 0x7ff7980b46b0 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.408+0000 7ff79659c640 10 monclient: get_auth_request con 0x7ff77806dbc0 auth_method 0 2024-03-29T10:55:02.391 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.408+0000 7ff786ffd640 10 client.4262.objecter ms_handle_connect 0x7ff77806dbc0 2024-03-29T10:55:02.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.416+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff788050360 con 0x7ff7980b46b0 2024-03-29T10:55:02.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.416+0000 7ff786ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:02.397 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.416+0000 7ff786ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:02.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.747+0000 7ff79ca7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.747+0000 7ff79ca7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.747+0000 7ff79ca7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:02.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.747+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff77c005330 con 0x7ff7980b46b0 2024-03-29T10:55:02.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.748+0000 7ff786ffd640 1 -- 172.21.15.22:0/913318967 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7ff788052460 con 0x7ff7980b46b0 2024-03-29T10:55:02.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.748+0000 7ff786ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:02.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.748+0000 7ff786ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:02.735 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:02.735 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.753+0000 7ff79ca7b640 10 librados: watch_flush enter 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 10 librados: watch_flush exit 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 20 client.4262.objecter shutdown clearing up homeless session... 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 10 client.4262.objecter successfully canceled tick 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 >> v1:172.21.15.22:6816/380221644 conn(0x7ff77806dbc0 legacy=0x7ff778070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 10 monclient: shutdown 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 >> v1:172.21.15.22:6790/0 conn(0x7ff7980b46b0 legacy=0x7ff7981384b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 shutdown_connections 2024-03-29T10:55:02.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.754+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 >> 172.21.15.22:0/913318967 conn(0x7ff79806d0f0 msgr2=0x7ff798137160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:02.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.755+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 shutdown_connections 2024-03-29T10:55:02.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.755+0000 7ff79ca7b640 1 -- 172.21.15.22:0/913318967 wait complete. 2024-03-29T10:55:02.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:02.755+0000 7ff79ca7b640 1 librados: shutdown 2024-03-29T10:55:02.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:09.560 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:09.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.732+0000 7f3becbd6640 10 monclient: get_monmap_and_config 2024-03-29T10:55:09.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.732+0000 7f3becbd6640 10 monclient: build_initial_monmap 2024-03-29T10:55:09.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.732+0000 7f3becbd6640 10 monclient: monmap: 2024-03-29T10:55:09.714 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:09.714 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:09.714 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:09.733282+0000 2024-03-29T10:55:09.714 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:09.733282+0000 2024-03-29T10:55:09.715 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:09.715 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:09.715 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:09.715 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:09.715 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:09.715 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:09.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.733+0000 7f3becbd6640 1 Processor -- start 2024-03-29T10:55:09.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.733+0000 7f3becbd6640 1 -- start start 2024-03-29T10:55:09.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.733+0000 7f3becbd6640 10 monclient: init 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f3be80a9ac0 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f3be813ee90 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f3be80af900 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80afd10 con 0x7f3be80a9ac0 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80af6f0 con 0x7f3be813ee90 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80aac80 con 0x7f3be80af900 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:09.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.734+0000 7f3becbd6640 10 monclient(hunting): authenticate will time out at 811.038932s 2024-03-29T10:55:09.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.738+0000 7f3be5d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f3be813ee90 0x7f3be813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:45717/0 (socket says 172.21.15.22:45717) 2024-03-29T10:55:09.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.738+0000 7f3be5d9b640 1 -- 172.21.15.22:0/4011039697 learned_addr learned my addr 172.21.15.22:0/4011039697 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:09.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109363686 0 0) 0x7f3be80af6f0 con 0x7f3be813ee90 2024-03-29T10:55:09.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): my global_id is 4264 2024-03-29T10:55:09.724 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd0003520 con 0x7f3be813ee90 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721468240 0 0) 0x7f3be80aac80 con 0x7f3be80af900 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 10 monclient(hunting): my global_id is 4265 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.742+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be80af6f0 con 0x7f3be80af900 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.743+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045378295 0 0) 0x7f3bd0003520 con 0x7f3be813ee90 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.743+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 >> v1:172.21.15.22:6790/0 conn(0x7f3be80af900 legacy=0x7f3be8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.743+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 >> v1:172.21.15.22:6789/0 conn(0x7f3be80a9ac0 legacy=0x7f3be80adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 1 monclient: found mon.noname-c 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be80aaf60 con 0x7f3be813ee90 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 10 monclient: _finish_auth 0 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:09.730 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3becbd6640 5 monclient: authenticate success, global_id 4264 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3becbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f3bdc001690 con 0x7f3be813ee90 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3be559a640 10 monclient: dump: 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:09.731 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.744+0000 7f3becbd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.746+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bdc0018b0 con 0x7f3be813ee90 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.746+0000 7f3be559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.746+0000 7f3be559a640 1 -- 172.21.15.22:0/4011039697 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f3bdc001c40 con 0x7f3be813ee90 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.746+0000 7f3be559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.747+0000 7f3be559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.747+0000 7f3be559a640 10 monclient: dump: 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.732 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.747+0000 7f3becbd6640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.747+0000 7f3becbd6640 4 set_mon_vals no callback set 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.747+0000 7f3becbd6640 10 monclient: shutdown 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.747+0000 7f3becbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.747+0000 7f3becbd6640 1 -- 172.21.15.22:0/4011039697 >> v1:172.21.15.26:6789/0 conn(0x7f3be813ee90 legacy=0x7f3be813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.748+0000 7f3becbd6640 1 -- 172.21.15.22:0/4011039697 shutdown_connections 2024-03-29T10:55:09.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.748+0000 7f3becbd6640 1 -- 172.21.15.22:0/4011039697 >> 172.21.15.22:0/4011039697 conn(0x7f3be80b35e0 msgr2=0x7f3be80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:09.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.749+0000 7f3becbd6640 1 -- 172.21.15.22:0/4011039697 shutdown_connections 2024-03-29T10:55:09.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.749+0000 7f3becbd6640 1 -- 172.21.15.22:0/4011039697 wait complete. 2024-03-29T10:55:09.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.750+0000 7f3becbd6640 10 monclient: build_initial_monmap 2024-03-29T10:55:09.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.750+0000 7f3becbd6640 10 monclient: monmap: 2024-03-29T10:55:09.736 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:09.736 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:09.736 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.750+0000 7f3becbd6640 1 librados: starting msgr at 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.750+0000 7f3becbd6640 1 librados: starting objecter 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.750+0000 7f3becbd6640 1 Processor -- start 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.751+0000 7f3becbd6640 1 -- start start 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.751+0000 7f3becbd6640 1 librados: setting wanted keys 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.751+0000 7f3becbd6640 1 librados: calling monclient init 2024-03-29T10:55:09.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.751+0000 7f3becbd6640 10 monclient: init 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.751+0000 7f3becbd6640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.751+0000 7f3becbd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 10 monclient(hunting): picked mon.noname-a con 0x7f3be80a9ac0 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 10 monclient(hunting): picked mon.noname-b con 0x7f3be80af900 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 10 monclient(hunting): picked mon.noname-c con 0x7f3be813ee90 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be813db40 con 0x7f3be80a9ac0 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be81dc790 con 0x7f3be80af900 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be813dfd0 con 0x7f3be813ee90 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.752+0000 7f3becbd6640 10 monclient(hunting): authenticate will time out at 811.056174s 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.755+0000 7f3be659c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f3be80a9ac0 0x7f3be81deb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48689/0 (socket says 172.21.15.22:48689) 2024-03-29T10:55:09.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.755+0000 7f3be659c640 1 -- 172.21.15.22:0/888456641 learned_addr learned my addr 172.21.15.22:0/888456641 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:09.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.756+0000 7f3bd6ffd640 10 client.?.objecter ms_handle_connect 0x7f3be80a9ac0 2024-03-29T10:55:09.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.756+0000 7f3bd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.756+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2125398028 0 0) 0x7f3be813db40 con 0x7f3be80a9ac0 2024-03-29T10:55:09.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.756+0000 7f3bd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.756+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.756+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.756+0000 7f3bd6ffd640 10 monclient(hunting): my global_id is 4266 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.757+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc8004eb0 con 0x7f3be80a9ac0 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.759+0000 7f3bd6ffd640 10 client.?.objecter ms_handle_connect 0x7f3be80af900 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.759+0000 7f3bd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.759+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2755412233 0 0) 0x7f3be81dc790 con 0x7f3be80af900 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.759+0000 7f3bd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.759+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.759+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.759+0000 7f3bd6ffd640 10 monclient(hunting): my global_id is 4267 2024-03-29T10:55:09.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be813db40 con 0x7f3be80af900 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2143326169 0 0) 0x7f3be813db40 con 0x7f3be80af900 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be81dc790 con 0x7f3be80af900 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f3bdc001bc0 con 0x7f3be80af900 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:09.743 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 client.?.objecter ms_handle_connect 0x7f3be813ee90 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1439645525 0 0) 0x7f3bc8004eb0 con 0x7f3be80a9ac0 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be813db40 con 0x7f3be80a9ac0 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f3be0001c90 con 0x7f3be80a9ac0 2024-03-29T10:55:09.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:09.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.760+0000 7f3bd6ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:09.749 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:09.749 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:09.749 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.749 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.749 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924542282 0 0) 0x7f3be813dfd0 con 0x7f3be813ee90 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient(hunting): my global_id is 4268 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc8004eb0 con 0x7f3be813ee90 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 828776890 0 0) 0x7f3be813db40 con 0x7f3be80a9ac0 2024-03-29T10:55:09.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 >> v1:172.21.15.22:6790/0 conn(0x7f3be813ee90 legacy=0x7f3be81dc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 >> v1:172.21.15.26:6789/0 conn(0x7f3be80af900 legacy=0x7f3be81db970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 1 monclient: found mon.a 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be813c0c0 con 0x7f3be80a9ac0 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3bd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 5 monclient: authenticate success, global_id 4266 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 10 monclient: _renew_subs 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be8124400 con 0x7f3be80a9ac0 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 10 client.4266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:09.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 10 monclient: _renew_subs 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be8123e20 con 0x7f3be80a9ac0 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.761+0000 7f3becbd6640 1 librados: init done 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.763+0000 7f3becbd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.763+0000 7f3becbd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.763+0000 7f3becbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.763+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be813bd00 con 0x7f3be80a9ac0 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.763+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3be0001df0 con 0x7f3be80a9ac0 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.763+0000 7f3bd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.763+0000 7f3bd7fff640 4 set_mon_vals no callback set 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.764+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f3be0003c30 con 0x7f3be80a9ac0 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.764+0000 7f3bd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.764+0000 7f3bd6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:09.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.764+0000 7f3bd6ffd640 10 monclient: dump: 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.764+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f3be0003e50 con 0x7f3be80a9ac0 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.765+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f3be0084de0 con 0x7f3be80a9ac0 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.765+0000 7f3be5d9b640 10 monclient: get_auth_request con 0x7f3bc806e030 auth_method 0 2024-03-29T10:55:09.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.766+0000 7f3bd6ffd640 10 client.4266.objecter ms_dispatch 0x7f3be8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:09.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.766+0000 7f3bd6ffd640 3 client.4266.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:09.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.766+0000 7f3bd6ffd640 3 client.4266.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:09.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.766+0000 7f3bd6ffd640 20 client.4266.objecter dump_active .. 0 homeless 2024-03-29T10:55:09.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.766+0000 7f3bd6ffd640 10 client.4266.objecter ms_handle_connect 0x7f3bc806e030 2024-03-29T10:55:09.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.773+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3be813bd00 con 0x7f3be80a9ac0 2024-03-29T10:55:09.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.774+0000 7f3bd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:09.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:09.774+0000 7f3bd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:10.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.111+0000 7f3becbd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:10.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.111+0000 7f3becbd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:10.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.111+0000 7f3becbd6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:10.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.111+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3be813bd00 con 0x7f3be80a9ac0 2024-03-29T10:55:10.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.112+0000 7f3bd6ffd640 1 -- 172.21.15.22:0/888456641 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f3be813bd00 con 0x7f3be80a9ac0 2024-03-29T10:55:10.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.112+0000 7f3bd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:10.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.112+0000 7f3bd6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:10.095 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:10.095 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:10.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.117+0000 7f3becbd6640 10 librados: watch_flush enter 2024-03-29T10:55:10.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.121+0000 7f3becbd6640 10 librados: watch_flush exit 2024-03-29T10:55:10.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.121+0000 7f3becbd6640 20 client.4266.objecter shutdown clearing up homeless session... 2024-03-29T10:55:10.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.121+0000 7f3becbd6640 10 client.4266.objecter successfully canceled tick 2024-03-29T10:55:10.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.122+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 >> v1:172.21.15.22:6816/380221644 conn(0x7f3bc806e030 legacy=0x7f3bc80704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:10.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.122+0000 7f3becbd6640 10 monclient: shutdown 2024-03-29T10:55:10.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.122+0000 7f3becbd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:10.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.122+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 >> v1:172.21.15.22:6789/0 conn(0x7f3be80a9ac0 legacy=0x7f3be81deb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:10.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.123+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 shutdown_connections 2024-03-29T10:55:10.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.123+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 >> 172.21.15.22:0/888456641 conn(0x7f3be80b35e0 msgr2=0x7f3be8133f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:10.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.124+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 shutdown_connections 2024-03-29T10:55:10.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.124+0000 7f3becbd6640 1 -- 172.21.15.22:0/888456641 wait complete. 2024-03-29T10:55:10.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:10.124+0000 7f3becbd6640 1 librados: shutdown 2024-03-29T10:55:10.122 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:16.985 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:17.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.154+0000 7fe4bb59e640 10 monclient: get_monmap_and_config 2024-03-29T10:55:17.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.154+0000 7fe4bb59e640 10 monclient: build_initial_monmap 2024-03-29T10:55:17.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.154+0000 7fe4bb59e640 10 monclient: monmap: 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:17.155467+0000 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:17.155467+0000 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:17.136 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:17.138 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:17.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.155+0000 7fe4bb59e640 1 Processor -- start 2024-03-29T10:55:17.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.155+0000 7fe4bb59e640 1 -- start start 2024-03-29T10:55:17.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.155+0000 7fe4bb59e640 10 monclient: init 2024-03-29T10:55:17.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.156+0000 7fe4bb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:17.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.156+0000 7fe4bb59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:17.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.160+0000 7fe4bb59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b40389b0 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:17.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.160+0000 7fe4bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b4038d90 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:17.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.163+0000 7fe4b9d9b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fe4b4038d90 0x7fe4b40cada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:42545/0 (socket says 172.21.15.22:42545) 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.163+0000 7fe4ba59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe4b40389b0 0x7fe4b40334f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:45095/0 (socket says 172.21.15.22:45095) 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.163+0000 7fe4bad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe4b4030630 0x7fe4b40d3140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:43729/0 (socket says 172.21.15.22:43729) 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.163+0000 7fe4bb59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b4030630 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.163+0000 7fe4bb59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40c5490 con 0x7fe4b4038d90 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.163+0000 7fe4b9d9b640 1 -- 172.21.15.22:0/1261821880 learned_addr learned my addr 172.21.15.22:0/1261821880 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1261821880 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b4036400 con 0x7fe4b40389b0 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1261821880 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b4039810 con 0x7fe4b4030630 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4bb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:17.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4bb59e640 10 monclient(hunting): authenticate will time out at 818.468116s 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951990392 0 0) 0x7fe4b4036400 con 0x7fe4b40389b0 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): my global_id is 4270 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4003520 con 0x7fe4b40389b0 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734204234 0 0) 0x7fe4b40c5490 con 0x7fe4b4038d90 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): my global_id is 4269 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b4036400 con 0x7fe4b4038d90 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846941820 0 0) 0x7fe4b4039810 con 0x7fe4b4030630 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 10 monclient(hunting): my global_id is 4271 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b40c5490 con 0x7fe4b4030630 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4172771081 0 0) 0x7fe4b4036400 con 0x7fe4b4038d90 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.164+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 >> v1:172.21.15.22:6790/0 conn(0x7fe4b4030630 legacy=0x7fe4b40d3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 >> v1:172.21.15.26:6789/0 conn(0x7fe4b40389b0 legacy=0x7fe4b40334f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 1 monclient: found mon.noname-a 2024-03-29T10:55:17.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b4035660 con 0x7fe4b4038d90 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 10 monclient: _finish_auth 0 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe4b0001670 con 0x7fe4b4038d90 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.165+0000 7fe4b959a640 10 monclient: dump: 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.151 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.167+0000 7fe4bb59e640 5 monclient: authenticate success, global_id 4269 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.167+0000 7fe4bb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4b00017d0 con 0x7fe4b4038d90 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4b959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4b959a640 1 -- 172.21.15.22:0/1261821880 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe4b0001be0 con 0x7fe4b4038d90 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4b959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4b959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4b959a640 10 monclient: dump: 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4bb59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:17.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.168+0000 7fe4bb59e640 4 set_mon_vals no callback set 2024-03-29T10:55:17.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.169+0000 7fe4bb59e640 10 monclient: shutdown 2024-03-29T10:55:17.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.169+0000 7fe4bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:17.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.169+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1261821880 >> v1:172.21.15.22:6789/0 conn(0x7fe4b4038d90 legacy=0x7fe4b40cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.169+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1261821880 shutdown_connections 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.169+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1261821880 >> 172.21.15.22:0/1261821880 conn(0x7fe4b4005bd0 msgr2=0x7fe4b4005fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.170+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1261821880 shutdown_connections 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.170+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1261821880 wait complete. 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.172+0000 7fe4bb59e640 10 monclient: build_initial_monmap 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.172+0000 7fe4bb59e640 10 monclient: monmap: 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:17.155 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:17.156 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:17.156 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:17.156 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:17.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.172+0000 7fe4bb59e640 1 librados: starting msgr at 2024-03-29T10:55:17.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.172+0000 7fe4bb59e640 1 librados: starting objecter 2024-03-29T10:55:17.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.172+0000 7fe4bb59e640 1 Processor -- start 2024-03-29T10:55:17.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.173+0000 7fe4bb59e640 1 -- start start 2024-03-29T10:55:17.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.173+0000 7fe4bb59e640 1 librados: setting wanted keys 2024-03-29T10:55:17.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.173+0000 7fe4bb59e640 1 librados: calling monclient init 2024-03-29T10:55:17.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.173+0000 7fe4bb59e640 10 monclient: init 2024-03-29T10:55:17.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.174+0000 7fe4bb59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:17.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.174+0000 7fe4bb59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:55:17.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.176+0000 7fe4bb59e640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b4030630 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:17.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.177+0000 7fe4bb59e640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b40389b0 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:17.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4ba59c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe4b4030630 0x7fe4b40c81e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:45239/0 (socket says 172.21.15.22:45239) 2024-03-29T10:55:17.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bb59e640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b415dbe0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:17.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bb59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b415d040 con 0x7fe4b40389b0 2024-03-29T10:55:17.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bb59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b40c7ac0 con 0x7fe4b4030630 2024-03-29T10:55:17.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bb59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b402fe20 con 0x7fe4b415dbe0 2024-03-29T10:55:17.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bb59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bb59e640 10 monclient(hunting): authenticate will time out at 818.484524s 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe4b415dbe0 0x7fe4b40bf7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:43843/0 (socket says 172.21.15.22:43843) 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4bad9d640 1 -- 172.21.15.22:0/1457995183 learned_addr learned my addr 172.21.15.22:0/1457995183 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4aaffd640 10 client.?.objecter ms_handle_connect 0x7fe4b415dbe0 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.180+0000 7fe4aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687788980 0 0) 0x7fe4b402fe20 con 0x7fe4b415dbe0 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): my global_id is 4274 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b415dbe0 2024-03-29T10:55:17.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2115256993 0 0) 0x7fe49c004eb0 con 0x7fe4b415dbe0 2024-03-29T10:55:17.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b402fe20 con 0x7fe4b415dbe0 2024-03-29T10:55:17.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 client.?.objecter ms_handle_connect 0x7fe4b40389b0 2024-03-29T10:55:17.165 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 client.?.objecter ms_handle_connect 0x7fe4b4030630 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe4bc04fdd0 con 0x7fe4b415dbe0 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): dump: 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:17.166 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710074397 0 0) 0x7fe4b415d040 con 0x7fe4b40389b0 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.181+0000 7fe4aaffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient(hunting): my global_id is 4272 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b40389b0 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1272479336 0 0) 0x7fe4b402fe20 con 0x7fe4b415dbe0 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 >> v1:172.21.15.26:6789/0 conn(0x7fe4b4030630 legacy=0x7fe4b40c81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 >> v1:172.21.15.22:6789/0 conn(0x7fe4b40389b0 legacy=0x7fe4b415d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 monclient: found mon.c 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b40c5620 con 0x7fe4b415dbe0 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 5 monclient: authenticate success, global_id 4274 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 10 monclient: _renew_subs 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b40c0480 con 0x7fe4b415dbe0 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 10 client.4274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:17.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 10 monclient: _renew_subs 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b40c0090 con 0x7fe4b415dbe0 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4bb59e640 1 librados: init done 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4bc04f400 con 0x7fe4b415dbe0 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe4bc075f10 con 0x7fe4b415dbe0 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.182+0000 7fe4aaffd640 10 monclient: dump: 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.169 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.184+0000 7fe4b8d99640 4 set_mon_vals no callback set 2024-03-29T10:55:17.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.184+0000 7fe4bb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:17.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.185+0000 7fe4bb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.185+0000 7fe4bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.185+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b4007b40 con 0x7fe4b415dbe0 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.186+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fe4bc067bf0 con 0x7fe4b415dbe0 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.187+0000 7fe4ba59c640 10 monclient: get_auth_request con 0x7fe49c06daf0 auth_method 0 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.187+0000 7fe4aaffd640 10 client.4274.objecter ms_handle_connect 0x7fe49c06daf0 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.187+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fe4bc0ff5a0 con 0x7fe4b415dbe0 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.187+0000 7fe4aaffd640 10 client.4274.objecter ms_dispatch 0x7fe4bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.187+0000 7fe4aaffd640 3 client.4274.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.188+0000 7fe4aaffd640 3 client.4274.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:17.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.188+0000 7fe4aaffd640 20 client.4274.objecter dump_active .. 0 homeless 2024-03-29T10:55:17.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.200+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4bc0cae80 con 0x7fe4b415dbe0 2024-03-29T10:55:17.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.200+0000 7fe4aaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:17.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.200+0000 7fe4aaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:17.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.536+0000 7fe4bb59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:17.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.536+0000 7fe4bb59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:17.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.536+0000 7fe4bb59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:17.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.536+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b40c57b0 con 0x7fe4b415dbe0 2024-03-29T10:55:17.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.537+0000 7fe4aaffd640 1 -- 172.21.15.22:0/1457995183 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fe4b40c57b0 con 0x7fe4b415dbe0 2024-03-29T10:55:17.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.537+0000 7fe4aaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:17.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.537+0000 7fe4aaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:17.522 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:17.522 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:17.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.544+0000 7fe4bb59e640 10 librados: watch_flush enter 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 10 librados: watch_flush exit 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 20 client.4274.objecter shutdown clearing up homeless session... 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 10 client.4274.objecter successfully canceled tick 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 >> v1:172.21.15.22:6816/380221644 conn(0x7fe49c06daf0 legacy=0x7fe49c06ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 10 monclient: shutdown 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 >> v1:172.21.15.22:6790/0 conn(0x7fe4b415dbe0 legacy=0x7fe4b40bf7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 shutdown_connections 2024-03-29T10:55:17.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.545+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 >> 172.21.15.22:0/1457995183 conn(0x7fe4b4005bd0 msgr2=0x7fe4b4034060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:17.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.546+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 shutdown_connections 2024-03-29T10:55:17.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.546+0000 7fe4bb59e640 1 -- 172.21.15.22:0/1457995183 wait complete. 2024-03-29T10:55:17.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:17.549+0000 7fe4bb59e640 1 librados: shutdown 2024-03-29T10:55:17.547 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:24.354 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:24.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.524+0000 7f230b9a8640 10 monclient: get_monmap_and_config 2024-03-29T10:55:24.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.524+0000 7f230b9a8640 10 monclient: build_initial_monmap 2024-03-29T10:55:24.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.525+0000 7f230b9a8640 10 monclient: monmap: 2024-03-29T10:55:24.506 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:24.506 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:24.506 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:24.525828+0000 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:24.525828+0000 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:24.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.525+0000 7f230b9a8640 1 Processor -- start 2024-03-29T10:55:24.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.526+0000 7f230b9a8640 1 -- start start 2024-03-29T10:55:24.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.526+0000 7f230b9a8640 10 monclient: init 2024-03-29T10:55:24.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:24.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:24.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f23040b0770 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:24.510 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f230414a700 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:24.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f23040ac5a0 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:24.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040aba70 con 0x7f23040b0770 2024-03-29T10:55:24.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040aecb0 con 0x7f23040ac5a0 2024-03-29T10:55:24.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23040af150 con 0x7f230414a700 2024-03-29T10:55:24.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:24.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.527+0000 7f230b9a8640 10 monclient(hunting): authenticate will time out at 825.831237s 2024-03-29T10:55:24.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.530+0000 7f2308f43640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f230414a700 0x7f23040abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58795/0 (socket says 172.21.15.22:58795) 2024-03-29T10:55:24.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.530+0000 7f2308f43640 1 -- 172.21.15.22:0/4269374456 learned_addr learned my addr 172.21.15.22:0/4269374456 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:24.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.532+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3164699398 0 0) 0x7f23040af150 con 0x7f230414a700 2024-03-29T10:55:24.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.532+0000 7f2303fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.532+0000 7f2303fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.532+0000 7f2303fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.532+0000 7f2303fff640 10 monclient(hunting): my global_id is 4277 2024-03-29T10:55:24.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.532+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ec003520 con 0x7f230414a700 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.535+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289372210 0 0) 0x7f23040aecb0 con 0x7f23040ac5a0 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.535+0000 7f2303fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.535+0000 7f2303fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient(hunting): my global_id is 4276 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23040af150 con 0x7f23040ac5a0 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620460614 0 0) 0x7f23040aba70 con 0x7f23040b0770 2024-03-29T10:55:24.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient(hunting): my global_id is 4275 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23040aecb0 con 0x7f23040b0770 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3096396678 0 0) 0x7f22ec003520 con 0x7f230414a700 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 >> v1:172.21.15.26:6789/0 conn(0x7f23040ac5a0 legacy=0x7f2304151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 >> v1:172.21.15.22:6789/0 conn(0x7f23040b0770 legacy=0x7f2304149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 monclient: found mon.noname-b 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23040ac9d0 con 0x7f230414a700 2024-03-29T10:55:24.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient: _finish_auth 0 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f22f4001990 con 0x7f230414a700 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f230b9a8640 5 monclient: authenticate success, global_id 4277 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f230b9a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f2303fff640 10 monclient: dump: 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.520 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:24.521 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:24.521 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:24.521 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:24.521 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:24.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.536+0000 7f230b9a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:24.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22f4001af0 con 0x7f230414a700 2024-03-29T10:55:24.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f2303fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:24.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f2303fff640 1 -- 172.21.15.22:0/4269374456 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f22f4001f00 con 0x7f230414a700 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f2303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f2303fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f2303fff640 10 monclient: dump: 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f230b9a8640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:24.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.540+0000 7f230b9a8640 4 set_mon_vals no callback set 2024-03-29T10:55:24.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.541+0000 7f230b9a8640 10 monclient: shutdown 2024-03-29T10:55:24.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.541+0000 7f230b9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:24.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.541+0000 7f230b9a8640 1 -- 172.21.15.22:0/4269374456 >> v1:172.21.15.22:6790/0 conn(0x7f230414a700 legacy=0x7f23040abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.541+0000 7f230b9a8640 1 -- 172.21.15.22:0/4269374456 shutdown_connections 2024-03-29T10:55:24.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.541+0000 7f230b9a8640 1 -- 172.21.15.22:0/4269374456 >> 172.21.15.22:0/4269374456 conn(0x7f230406b660 msgr2=0x7f230406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:24.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.542+0000 7f230b9a8640 1 -- 172.21.15.22:0/4269374456 shutdown_connections 2024-03-29T10:55:24.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.542+0000 7f230b9a8640 1 -- 172.21.15.22:0/4269374456 wait complete. 2024-03-29T10:55:24.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.542+0000 7f230b9a8640 10 monclient: build_initial_monmap 2024-03-29T10:55:24.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.542+0000 7f230b9a8640 10 monclient: monmap: 2024-03-29T10:55:24.526 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:24.526 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:24.526 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:24.526 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.543+0000 7f230b9a8640 1 librados: starting msgr at 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.543+0000 7f230b9a8640 1 librados: starting objecter 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.543+0000 7f230b9a8640 1 Processor -- start 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.543+0000 7f230b9a8640 1 -- start start 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.543+0000 7f230b9a8640 1 librados: setting wanted keys 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.543+0000 7f230b9a8640 1 librados: calling monclient init 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.543+0000 7f230b9a8640 10 monclient: init 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:24.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 10 monclient(hunting): picked mon.noname-c con 0x7f23040ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 10 monclient(hunting): picked mon.noname-b con 0x7f23040b0770 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 10 monclient(hunting): picked mon.noname-a con 0x7f230414a700 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2304128400 con 0x7f230414a700 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2304126280 con 0x7f23040b0770 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2304125da0 con 0x7f23040ac5a0 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:24.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.544+0000 7f230b9a8640 10 monclient(hunting): authenticate will time out at 825.848629s 2024-03-29T10:55:24.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.550+0000 7f2308f43640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f23040b0770 0x7f23041267a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:54939/0 (socket says 172.21.15.22:54939) 2024-03-29T10:55:24.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.550+0000 7f2308f43640 1 -- 172.21.15.22:0/2331020843 learned_addr learned my addr 172.21.15.22:0/2331020843 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:24.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.552+0000 7f2301ffb640 10 client.?.objecter ms_handle_connect 0x7f230414a700 2024-03-29T10:55:24.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.552+0000 7f2301ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 client.?.objecter ms_handle_connect 0x7f23040b0770 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155015254 0 0) 0x7f2304128400 con 0x7f230414a700 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): my global_id is 4278 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0004eb0 con 0x7f230414a700 2024-03-29T10:55:24.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527742001 0 0) 0x7f2304126280 con 0x7f23040b0770 2024-03-29T10:55:24.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 10 monclient(hunting): my global_id is 4279 2024-03-29T10:55:24.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.554+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2304128400 con 0x7f23040b0770 2024-03-29T10:55:24.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.555+0000 7f2301ffb640 10 client.?.objecter ms_handle_connect 0x7f23040ac5a0 2024-03-29T10:55:24.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.555+0000 7f2301ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:24.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 303184037 0 0) 0x7f22f0004eb0 con 0x7f230414a700 2024-03-29T10:55:24.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2304126280 con 0x7f230414a700 2024-03-29T10:55:24.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012778363 0 0) 0x7f2304125da0 con 0x7f23040ac5a0 2024-03-29T10:55:24.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:24.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:24.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:24.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): my global_id is 4280 2024-03-29T10:55:24.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0004eb0 con 0x7f23040ac5a0 2024-03-29T10:55:24.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 496277518 0 0) 0x7f2304128400 con 0x7f23040b0770 2024-03-29T10:55:24.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2304125da0 con 0x7f23040b0770 2024-03-29T10:55:24.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f22fc0012d0 con 0x7f230414a700 2024-03-29T10:55:24.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:24.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:24.548 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:24.548 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:24.548 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.549 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.549 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.549 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:24.549 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:24.549 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:24.549 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:24.550 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:24.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:24.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f22f4001e00 con 0x7f23040b0770 2024-03-29T10:55:24.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:24.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.558+0000 7f2301ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:24.551 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:24.551 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:24.551 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.551 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.551 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.551 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:24.552 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:24.552 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:24.552 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:24.552 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:24.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2139942622 0 0) 0x7f2304125da0 con 0x7f23040b0770 2024-03-29T10:55:24.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 >> v1:172.21.15.22:6790/0 conn(0x7f23040ac5a0 legacy=0x7f2304144fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 >> v1:172.21.15.22:6789/0 conn(0x7f230414a700 legacy=0x7f23041e6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:24.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 1 monclient: found mon.b 2024-03-29T10:55:24.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:24.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:24.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2304125f30 con 0x7f23040b0770 2024-03-29T10:55:24.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:24.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:24.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.559+0000 7f2301ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:24.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 5 monclient: authenticate success, global_id 4279 2024-03-29T10:55:24.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 10 monclient: _renew_subs 2024-03-29T10:55:24.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:24.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23041dc1c0 con 0x7f23040b0770 2024-03-29T10:55:24.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 10 client.4279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:24.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 10 monclient: _renew_subs 2024-03-29T10:55:24.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:24.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23041458b0 con 0x7f23040b0770 2024-03-29T10:55:24.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.560+0000 7f230b9a8640 1 librados: init done 2024-03-29T10:55:24.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.562+0000 7f230b9a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:24.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.562+0000 7f230b9a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:24.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.562+0000 7f230b9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:24.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.562+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2304128240 con 0x7f23040b0770 2024-03-29T10:55:24.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.563+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22f4002000 con 0x7f23040b0770 2024-03-29T10:55:24.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.563+0000 7f2301ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:24.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.563+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f22f4004e10 con 0x7f23040b0770 2024-03-29T10:55:24.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.563+0000 7f2301ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:24.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.563+0000 7f2301ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:24.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.563+0000 7f2301ffb640 10 monclient: dump: 2024-03-29T10:55:24.557 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:24.557 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:24.558 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:24.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.563+0000 7f2302ffd640 4 set_mon_vals no callback set 2024-03-29T10:55:24.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.567+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f22f4004f70 con 0x7f23040b0770 2024-03-29T10:55:24.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.567+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f22f4084c40 con 0x7f23040b0770 2024-03-29T10:55:24.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.567+0000 7f2301ffb640 10 client.4279.objecter ms_dispatch 0x7f2304000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:24.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.567+0000 7f2301ffb640 3 client.4279.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:24.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.567+0000 7f2301ffb640 3 client.4279.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:24.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.568+0000 7f2301ffb640 20 client.4279.objecter dump_active .. 0 homeless 2024-03-29T10:55:24.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.568+0000 7f2309744640 10 monclient: get_auth_request con 0x7f22f006dfd0 auth_method 0 2024-03-29T10:55:24.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.568+0000 7f2301ffb640 10 client.4279.objecter ms_handle_connect 0x7f22f006dfd0 2024-03-29T10:55:24.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.572+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22f4050520 con 0x7f23040b0770 2024-03-29T10:55:24.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.572+0000 7f2301ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:24.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.572+0000 7f2301ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:24.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.893+0000 7f230b9a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.893+0000 7f230b9a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.893+0000 7f230b9a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:24.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.893+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2304126410 con 0x7f23040b0770 2024-03-29T10:55:24.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.895+0000 7f2301ffb640 1 -- 172.21.15.22:0/2331020843 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f22f4052450 con 0x7f23040b0770 2024-03-29T10:55:24.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.895+0000 7f2301ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:24.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.895+0000 7f2301ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:24.876 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:24.876 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:24.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 10 librados: watch_flush enter 2024-03-29T10:55:24.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 10 librados: watch_flush exit 2024-03-29T10:55:24.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 20 client.4279.objecter shutdown clearing up homeless session... 2024-03-29T10:55:24.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 10 client.4279.objecter successfully canceled tick 2024-03-29T10:55:24.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 >> v1:172.21.15.22:6816/380221644 conn(0x7f22f006dfd0 legacy=0x7f22f0070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 10 monclient: shutdown 2024-03-29T10:55:24.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:24.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 >> v1:172.21.15.26:6789/0 conn(0x7f23040b0770 legacy=0x7f23041267a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:24.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 shutdown_connections 2024-03-29T10:55:24.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.898+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 >> 172.21.15.22:0/2331020843 conn(0x7f230406b660 msgr2=0x7f23040b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:24.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.901+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 shutdown_connections 2024-03-29T10:55:24.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.901+0000 7f230b9a8640 1 -- 172.21.15.22:0/2331020843 wait complete. 2024-03-29T10:55:24.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:24.901+0000 7f230b9a8640 1 librados: shutdown 2024-03-29T10:55:24.902 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:31.709 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:31.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.909+0000 7f75c7fff640 10 monclient: get_monmap_and_config 2024-03-29T10:55:31.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.909+0000 7f75c7fff640 10 monclient: build_initial_monmap 2024-03-29T10:55:31.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.909+0000 7f75c7fff640 10 monclient: monmap: 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:31.910569+0000 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:31.910569+0000 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:31.891 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:31.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.910+0000 7f75c7fff640 1 Processor -- start 2024-03-29T10:55:31.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.913+0000 7f75c7fff640 1 -- start start 2024-03-29T10:55:31.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.913+0000 7f75c7fff640 10 monclient: init 2024-03-29T10:55:31.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:31.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:55:31.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f75c80b2800 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:31.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f75c8142b20 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:31.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f75c813c040 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:31.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c80aad10 con 0x7f75c8142b20 2024-03-29T10:55:31.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c8141f30 con 0x7f75c813c040 2024-03-29T10:55:31.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c80aaf60 con 0x7f75c80b2800 2024-03-29T10:55:31.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:31.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.914+0000 7f75c7fff640 10 monclient(hunting): authenticate will time out at 833.219019s 2024-03-29T10:55:31.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.919+0000 7f75c6ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f75c80b2800 0x7f75c8142410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:33303/0 (socket says 172.21.15.22:33303) 2024-03-29T10:55:31.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.919+0000 7f75c6ffd640 1 -- 172.21.15.22:0/2218902968 learned_addr learned my addr 172.21.15.22:0/2218902968 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:31.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.921+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4163379562 0 0) 0x7f75c80aaf60 con 0x7f75c80b2800 2024-03-29T10:55:31.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.921+0000 7f75c67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.921+0000 7f75c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.921+0000 7f75c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.921+0000 7f75c67fc640 10 monclient(hunting): my global_id is 4283 2024-03-29T10:55:31.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.921+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b4003520 con 0x7f75c80b2800 2024-03-29T10:55:31.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.922+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1283148393 0 0) 0x7f75b4003520 con 0x7f75c80b2800 2024-03-29T10:55:31.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.922+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 >> v1:172.21.15.26:6789/0 conn(0x7f75c813c040 legacy=0x7f75c814d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 >> v1:172.21.15.22:6789/0 conn(0x7f75c8142b20 legacy=0x7f75c813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:31.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:31.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:31.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:55:31.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c80b3900 con 0x7f75c80b2800 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f75b8001690 con 0x7f75c80b2800 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c67fc640 10 monclient: dump: 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:31.905 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c7fff640 5 monclient: authenticate success, global_id 4283 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.923+0000 7f75c7fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.925+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b80017f0 con 0x7f75c80b2800 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.926+0000 7f75c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.926+0000 7f75c67fc640 1 -- 172.21.15.22:0/2218902968 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f75b8001c00 con 0x7f75c80b2800 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.926+0000 7f75c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.926+0000 7f75c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.926+0000 7f75c67fc640 10 monclient: dump: 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:31.907 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:31.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.927+0000 7f75c7fff640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:31.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.927+0000 7f75c7fff640 4 set_mon_vals no callback set 2024-03-29T10:55:31.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.927+0000 7f75c7fff640 10 monclient: shutdown 2024-03-29T10:55:31.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.927+0000 7f75c7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:31.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.927+0000 7f75c7fff640 1 -- 172.21.15.22:0/2218902968 >> v1:172.21.15.22:6790/0 conn(0x7f75c80b2800 legacy=0x7f75c8142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.928+0000 7f75c7fff640 1 -- 172.21.15.22:0/2218902968 shutdown_connections 2024-03-29T10:55:31.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.928+0000 7f75c7fff640 1 -- 172.21.15.22:0/2218902968 >> 172.21.15.22:0/2218902968 conn(0x7f75c806d0e0 msgr2=0x7f75c8066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:31.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.928+0000 7f75c7fff640 1 -- 172.21.15.22:0/2218902968 shutdown_connections 2024-03-29T10:55:31.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.933+0000 7f75c7fff640 1 -- 172.21.15.22:0/2218902968 wait complete. 2024-03-29T10:55:31.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.934+0000 7f75c7fff640 10 monclient: build_initial_monmap 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.934+0000 7f75c7fff640 10 monclient: monmap: 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:31.915 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:31.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.934+0000 7f75c7fff640 1 librados: starting msgr at 2024-03-29T10:55:31.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.934+0000 7f75c7fff640 1 librados: starting objecter 2024-03-29T10:55:31.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.935+0000 7f75c7fff640 1 Processor -- start 2024-03-29T10:55:31.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.935+0000 7f75c7fff640 1 -- start start 2024-03-29T10:55:31.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.935+0000 7f75c7fff640 1 librados: setting wanted keys 2024-03-29T10:55:31.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.935+0000 7f75c7fff640 1 librados: calling monclient init 2024-03-29T10:55:31.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.935+0000 7f75c7fff640 10 monclient: init 2024-03-29T10:55:31.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f75c80b2800 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 10 monclient(hunting): picked mon.noname-c con 0x7f75c813c040 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f75c8142b20 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c81449b0 con 0x7f75c80b2800 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c81d6d40 con 0x7f75c8142b20 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c81d7970 con 0x7f75c813c040 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.936+0000 7f75c7fff640 10 monclient(hunting): authenticate will time out at 833.240937s 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.940+0000 7f75c6ffd640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f75c80b2800 0x7f75c8143200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:39213/0 (socket says 172.21.15.22:39213) 2024-03-29T10:55:31.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.940+0000 7f75c6ffd640 1 -- 172.21.15.22:0/1044929717 learned_addr learned my addr 172.21.15.22:0/1044929717 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.941+0000 7f75bffff640 10 client.?.objecter ms_handle_connect 0x7f75c80b2800 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.941+0000 7f75bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641031736 0 0) 0x7f75c81449b0 con 0x7f75c80b2800 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 10 monclient(hunting): my global_id is 4284 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a8004eb0 con 0x7f75c80b2800 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 10 client.?.objecter ms_handle_connect 0x7f75c8142b20 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.942+0000 7f75bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:31.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 client.?.objecter ms_handle_connect 0x7f75c813c040 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1800852073 0 0) 0x7f75a8004eb0 con 0x7f75c80b2800 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75c81449b0 con 0x7f75c80b2800 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f75b8001bc0 con 0x7f75c80b2800 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): dump: 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.926 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875352552 0 0) 0x7f75c81d7970 con 0x7f75c813c040 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 10 monclient(hunting): my global_id is 4286 2024-03-29T10:55:31.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.943+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75a8004eb0 con 0x7f75c813c040 2024-03-29T10:55:31.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171000388 0 0) 0x7f75c81d6d40 con 0x7f75c8142b20 2024-03-29T10:55:31.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient(hunting): my global_id is 4285 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c81d7970 con 0x7f75c8142b20 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2225146126 0 0) 0x7f75c81449b0 con 0x7f75c80b2800 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 >> v1:172.21.15.22:6790/0 conn(0x7f75c813c040 legacy=0x7f75c81d7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 >> v1:172.21.15.26:6789/0 conn(0x7f75c8142b20 legacy=0x7f75c8136d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 1 monclient: found mon.a 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c81d7b00 con 0x7f75c80b2800 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient: _finish_auth 0 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:31.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.945+0000 7f75bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.948+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75b8004a90 con 0x7f75c80b2800 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.948+0000 7f75bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.948+0000 7f75c5ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f75b8004de0 con 0x7f75c80b2800 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75bffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75bffff640 10 monclient: dump: 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:31.930 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 5 monclient: authenticate success, global_id 4284 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 10 monclient: _renew_subs 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c8137440 con 0x7f75c80b2800 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 10 client.4284.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 10 monclient: _renew_subs 2024-03-29T10:55:31.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:31.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c8144d60 con 0x7f75c80b2800 2024-03-29T10:55:31.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.949+0000 7f75c7fff640 1 librados: init done 2024-03-29T10:55:31.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.950+0000 7f75c7fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.950+0000 7f75c7fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.950+0000 7f75c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:31.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.951+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75c804f730 con 0x7f75c80b2800 2024-03-29T10:55:31.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.952+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f75b801adb0 con 0x7f75c80b2800 2024-03-29T10:55:31.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.957+0000 7f75be7fc640 10 monclient: get_auth_request con 0x7f75a806dd80 auth_method 0 2024-03-29T10:55:31.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.957+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f75b8054980 con 0x7f75c80b2800 2024-03-29T10:55:31.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.957+0000 7f75bffff640 10 client.4284.objecter ms_dispatch 0x7f75c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:31.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.957+0000 7f75bffff640 3 client.4284.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:31.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.957+0000 7f75bffff640 3 client.4284.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:31.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.957+0000 7f75bffff640 20 client.4284.objecter dump_active .. 0 homeless 2024-03-29T10:55:31.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.957+0000 7f75bffff640 10 client.4284.objecter ms_handle_connect 0x7f75a806dd80 2024-03-29T10:55:31.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.964+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f75b8050490 con 0x7f75c80b2800 2024-03-29T10:55:31.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.964+0000 7f75bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:31.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:31.964+0000 7f75bffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:32.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.325+0000 7f75c7fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.326+0000 7f75c7fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.326+0000 7f75c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:55:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.326+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75c81d6ed0 con 0x7f75c80b2800 2024-03-29T10:55:32.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.327+0000 7f75bffff640 1 -- 172.21.15.22:0/1044929717 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f75b80523c0 con 0x7f75c80b2800 2024-03-29T10:55:32.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.327+0000 7f75bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:32.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.327+0000 7f75bffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:32.308 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:32.308 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:32.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 10 librados: watch_flush enter 2024-03-29T10:55:32.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 10 librados: watch_flush exit 2024-03-29T10:55:32.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 20 client.4284.objecter shutdown clearing up homeless session... 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 10 client.4284.objecter successfully canceled tick 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 >> v1:172.21.15.22:6816/380221644 conn(0x7f75a806dd80 legacy=0x7f75a8070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 10 monclient: shutdown 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 >> v1:172.21.15.22:6789/0 conn(0x7f75c80b2800 legacy=0x7f75c8143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 shutdown_connections 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 >> 172.21.15.22:0/1044929717 conn(0x7f75c806d0e0 msgr2=0x7f75c80b4360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.330+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 shutdown_connections 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.331+0000 7f75c7fff640 1 -- 172.21.15.22:0/1044929717 wait complete. 2024-03-29T10:55:32.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:32.331+0000 7f75c7fff640 1 librados: shutdown 2024-03-29T10:55:32.327 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:39.134 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.299+0000 7f638b417640 10 monclient: get_monmap_and_config 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.299+0000 7f638b417640 10 monclient: build_initial_monmap 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.299+0000 7f638b417640 10 monclient: monmap: 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:39.300391+0000 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:39.300391+0000 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:39.280 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:39.281 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:39.281 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:39.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.300+0000 7f638b417640 1 Processor -- start 2024-03-29T10:55:39.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.301+0000 7f638b417640 1 -- start start 2024-03-29T10:55:39.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.301+0000 7f638b417640 10 monclient: init 2024-03-29T10:55:39.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:39.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 10 monclient(hunting): picked mon.noname-a con 0x7f638406c250 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 10 monclient(hunting): picked mon.noname-c con 0x7f6384072400 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 10 monclient(hunting): picked mon.noname-b con 0x7f638406c630 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f638406d760 con 0x7f638406c250 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6384070500 con 0x7f6384072400 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63840aff50 con 0x7f638406c630 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f63899b4640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f638406c630 0x7f6384151ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39144/0 (socket says 172.21.15.22:39144) 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f63899b4640 1 -- 172.21.15.22:0/1207313474 learned_addr learned my addr 172.21.15.22:0/1207313474 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:39.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.302+0000 7f638b417640 10 monclient(hunting): authenticate will time out at 840.606604s 2024-03-29T10:55:39.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684075041 0 0) 0x7f63840aff50 con 0x7f638406c630 2024-03-29T10:55:39.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:39.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:39.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:39.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): my global_id is 4289 2024-03-29T10:55:39.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6370003520 con 0x7f638406c630 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9490386 0 0) 0x7f638406d760 con 0x7f638406c250 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): my global_id is 4287 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63840aff50 con 0x7f638406c250 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338400159 0 0) 0x7f6384070500 con 0x7f6384072400 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 10 monclient(hunting): my global_id is 4288 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.303+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f638406d760 con 0x7f6384072400 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1358365487 0 0) 0x7f6370003520 con 0x7f638406c630 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 >> v1:172.21.15.26:6789/0 conn(0x7f6384072400 legacy=0x7f63841465b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 >> v1:172.21.15.22:6789/0 conn(0x7f638406c250 legacy=0x7f6384071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 1 monclient: found mon.noname-b 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63840b00e0 con 0x7f638406c630 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 10 monclient: _finish_auth 0 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f637c001a10 con 0x7f638406c630 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f637bfff640 10 monclient: dump: 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:39.286 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f638b417640 5 monclient: authenticate success, global_id 4289 2024-03-29T10:55:39.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.304+0000 7f638b417640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:39.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f637c001b70 con 0x7f638406c630 2024-03-29T10:55:39.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f637bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:39.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f637bfff640 1 -- 172.21.15.22:0/1207313474 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f637c001f80 con 0x7f638406c630 2024-03-29T10:55:39.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f637bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:39.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f637bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:39.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f637bfff640 10 monclient: dump: 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f638b417640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.305+0000 7f638b417640 4 set_mon_vals no callback set 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.306+0000 7f638b417640 10 monclient: shutdown 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.306+0000 7f638b417640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:39.289 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.306+0000 7f638b417640 1 -- 172.21.15.22:0/1207313474 >> v1:172.21.15.22:6790/0 conn(0x7f638406c630 legacy=0x7f6384151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.306+0000 7f638b417640 1 -- 172.21.15.22:0/1207313474 shutdown_connections 2024-03-29T10:55:39.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.306+0000 7f638b417640 1 -- 172.21.15.22:0/1207313474 >> 172.21.15.22:0/1207313474 conn(0x7f6384067610 msgr2=0x7f63840b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:39.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.307+0000 7f638b417640 1 -- 172.21.15.22:0/1207313474 shutdown_connections 2024-03-29T10:55:39.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.307+0000 7f638b417640 1 -- 172.21.15.22:0/1207313474 wait complete. 2024-03-29T10:55:39.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.308+0000 7f638b417640 10 monclient: build_initial_monmap 2024-03-29T10:55:39.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.308+0000 7f638b417640 10 monclient: monmap: 2024-03-29T10:55:39.293 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:39.293 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.308+0000 7f638b417640 1 librados: starting msgr at 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.308+0000 7f638b417640 1 librados: starting objecter 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.309+0000 7f638b417640 1 Processor -- start 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.309+0000 7f638b417640 1 -- start start 2024-03-29T10:55:39.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.309+0000 7f638b417640 1 librados: setting wanted keys 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.309+0000 7f638b417640 1 librados: calling monclient init 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.309+0000 7f638b417640 10 monclient: init 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 10 monclient(hunting): picked mon.noname-c con 0x7f638406c250 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 10 monclient(hunting): picked mon.noname-b con 0x7f638406c630 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 10 monclient(hunting): picked mon.noname-a con 0x7f6384072400 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6384141040 con 0x7f6384072400 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6384142d20 con 0x7f638406c630 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63840b3610 con 0x7f638406c250 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f638b417640 10 monclient(hunting): authenticate will time out at 840.614679s 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f63889b2640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f638406c630 0x7f63841df080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:54530/0 (socket says 172.21.15.22:54530) 2024-03-29T10:55:39.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f63889b2640 1 -- 172.21.15.22:0/4200739260 learned_addr learned my addr 172.21.15.22:0/4200739260 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f6379ffb640 10 client.?.objecter ms_handle_connect 0x7f6384072400 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f6379ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f6379ffb640 10 client.?.objecter ms_handle_connect 0x7f638406c630 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.310+0000 7f6379ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2874020400 0 0) 0x7f6384141040 con 0x7f6384072400 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): my global_id is 4290 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6368004eb0 con 0x7f6384072400 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852661856 0 0) 0x7f6384142d20 con 0x7f638406c630 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:39.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 monclient(hunting): my global_id is 4291 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6384141040 con 0x7f638406c630 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 client.?.objecter ms_handle_connect 0x7f638406c250 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.311+0000 7f6379ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215049253 0 0) 0x7f6384141040 con 0x7f638406c630 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6384142d20 con 0x7f638406c630 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6374001950 con 0x7f638406c630 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): dump: 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:39.297 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124289483 0 0) 0x7f63840b3610 con 0x7f638406c250 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient(hunting): my global_id is 4292 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6384141040 con 0x7f638406c250 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4247789194 0 0) 0x7f6384142d20 con 0x7f638406c630 2024-03-29T10:55:39.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 >> v1:172.21.15.22:6790/0 conn(0x7f638406c250 legacy=0x7f6384140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 >> v1:172.21.15.22:6789/0 conn(0x7f6384072400 legacy=0x7f63840ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 monclient: found mon.b 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63841df790 con 0x7f638406c630 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient: _finish_auth 0 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f6379ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f638b417640 5 monclient: authenticate success, global_id 4291 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f638b417640 10 monclient: _renew_subs 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f638b417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.312+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63841dfa40 con 0x7f638406c630 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f638b417640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:39.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f638b417640 10 monclient: _renew_subs 2024-03-29T10:55:39.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f638b417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:39.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63840af130 con 0x7f638406c630 2024-03-29T10:55:39.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f638b417640 1 librados: init done 2024-03-29T10:55:39.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6374003610 con 0x7f638406c630 2024-03-29T10:55:39.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f6379ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:39.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6374003a20 con 0x7f638406c630 2024-03-29T10:55:39.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f6379ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:39.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f637b7fe640 4 set_mon_vals no callback set 2024-03-29T10:55:39.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f6379ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.313+0000 7f6379ffb640 10 monclient: dump: 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.314+0000 7f638b417640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.315+0000 7f638b417640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.315+0000 7f638b417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:39.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.315+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6364005080 con 0x7f638406c630 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.316+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f6374003b80 con 0x7f638406c630 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.317+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f6374084a00 con 0x7f638406c630 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.317+0000 7f6379ffb640 10 client.4291.objecter ms_dispatch 0x7f6384000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.317+0000 7f6379ffb640 3 client.4291.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.317+0000 7f6379ffb640 3 client.4291.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.317+0000 7f6379ffb640 20 client.4291.objecter dump_active .. 0 homeless 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.317+0000 7f63891b3640 10 monclient: get_auth_request con 0x7f636806de10 auth_method 0 2024-03-29T10:55:39.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.318+0000 7f6379ffb640 10 client.4291.objecter ms_handle_connect 0x7f636806de10 2024-03-29T10:55:39.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.323+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6374050340 con 0x7f638406c630 2024-03-29T10:55:39.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.323+0000 7f6379ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:39.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.323+0000 7f6379ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:39.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.753+0000 7f638b417640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:39.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.753+0000 7f638b417640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:39.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.753+0000 7f638b417640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:55:39.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.753+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6364005330 con 0x7f638406c630 2024-03-29T10:55:39.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.754+0000 7f6379ffb640 1 -- 172.21.15.22:0/4200739260 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f63740840b0 con 0x7f638406c630 2024-03-29T10:55:39.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.754+0000 7f6379ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:39.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.754+0000 7f6379ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:39.735 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:39.735 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.759+0000 7f638b417640 10 librados: watch_flush enter 2024-03-29T10:55:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.759+0000 7f638b417640 10 librados: watch_flush exit 2024-03-29T10:55:39.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.759+0000 7f638b417640 20 client.4291.objecter shutdown clearing up homeless session... 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.759+0000 7f638b417640 10 client.4291.objecter successfully canceled tick 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 >> v1:172.21.15.22:6816/380221644 conn(0x7f636806de10 legacy=0x7f63680702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 10 monclient: shutdown 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 >> v1:172.21.15.26:6789/0 conn(0x7f638406c630 legacy=0x7f63841df080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 shutdown_connections 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 >> 172.21.15.22:0/4200739260 conn(0x7f6384067610 msgr2=0x7f6384149ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 shutdown_connections 2024-03-29T10:55:39.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 1 -- 172.21.15.22:0/4200739260 wait complete. 2024-03-29T10:55:39.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:39.760+0000 7f638b417640 1 librados: shutdown 2024-03-29T10:55:39.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:46.563 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:46.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.715+0000 7f995cd6c640 10 monclient: get_monmap_and_config 2024-03-29T10:55:46.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.715+0000 7f995cd6c640 10 monclient: build_initial_monmap 2024-03-29T10:55:46.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.716+0000 7f995cd6c640 10 monclient: monmap: 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:46.716791+0000 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:46.716791+0000 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:46.697 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.716+0000 7f995cd6c640 1 Processor -- start 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.716+0000 7f995cd6c640 1 -- start start 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.716+0000 7f995cd6c640 10 monclient: init 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 10 monclient(hunting): picked mon.noname-b con 0x7f99581457f0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 10 monclient(hunting): picked mon.noname-c con 0x7f995813f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 10 monclient(hunting): picked mon.noname-a con 0x7f99580a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:46.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580aaef0 con 0x7f99580a9d70 2024-03-29T10:55:46.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958145bd0 con 0x7f995813f650 2024-03-29T10:55:46.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.717+0000 7f995cd6c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958145eb0 con 0x7f99581457f0 2024-03-29T10:55:46.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.718+0000 7f995cd6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:46.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.718+0000 7f995cd6c640 10 monclient(hunting): authenticate will time out at 848.022119s 2024-03-29T10:55:46.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.718+0000 7f995659c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f99581457f0 0x7f995813ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:43872/0 (socket says 172.21.15.22:43872) 2024-03-29T10:55:46.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.718+0000 7f995659c640 1 -- 172.21.15.22:0/1031409929 learned_addr learned my addr 172.21.15.22:0/1031409929 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:46.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328403110 0 0) 0x7f9958145eb0 con 0x7f99581457f0 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 10 monclient(hunting): _negotiate 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 10 monclient(hunting): my global_id is 4295 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9940003520 con 0x7f99581457f0 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820068697 0 0) 0x7f9940003520 con 0x7f99581457f0 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 >> v1:172.21.15.26:6789/0 conn(0x7f995813f650 legacy=0x7f99580b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:46.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.719+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 >> v1:172.21.15.22:6789/0 conn(0x7f99580a9d70 legacy=0x7f995814d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: _finish_hunting 0 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 1 monclient: found mon.noname-b 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958146340 con 0x7f99581457f0 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: _finish_auth 0 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: _check_auth_tickets 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9948001690 con 0x7f99581457f0 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: dump: 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995cd6c640 5 monclient: authenticate success, global_id 4295 2024-03-29T10:55:46.703 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995cd6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99480018f0 con 0x7f99581457f0 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995cd6c640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.721+0000 7f995cd6c640 4 set_mon_vals no callback set 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.722+0000 7f995559a640 1 -- 172.21.15.22:0/1031409929 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9948001d40 con 0x7f99581457f0 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.722+0000 7f995559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.722+0000 7f995559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:46.706 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.722+0000 7f995559a640 10 monclient: dump: 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.722+0000 7f995cd6c640 10 monclient: shutdown 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.722+0000 7f995cd6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.722+0000 7f995cd6c640 1 -- 172.21.15.22:0/1031409929 >> v1:172.21.15.22:6790/0 conn(0x7f99581457f0 legacy=0x7f995813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:46.707 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.723+0000 7f995cd6c640 1 -- 172.21.15.22:0/1031409929 shutdown_connections 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.723+0000 7f995cd6c640 1 -- 172.21.15.22:0/1031409929 >> 172.21.15.22:0/1031409929 conn(0x7f9958066a00 msgr2=0x7f9958066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.723+0000 7f995cd6c640 1 -- 172.21.15.22:0/1031409929 shutdown_connections 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.723+0000 7f995cd6c640 1 -- 172.21.15.22:0/1031409929 wait complete. 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.724+0000 7f995cd6c640 10 monclient: build_initial_monmap 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.724+0000 7f995cd6c640 10 monclient: monmap: 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:46.708 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:46.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.724+0000 7f995cd6c640 1 librados: starting msgr at 2024-03-29T10:55:46.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.725+0000 7f995cd6c640 1 librados: starting objecter 2024-03-29T10:55:46.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.725+0000 7f995cd6c640 1 Processor -- start 2024-03-29T10:55:46.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.725+0000 7f995cd6c640 1 -- start start 2024-03-29T10:55:46.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.725+0000 7f995cd6c640 1 librados: setting wanted keys 2024-03-29T10:55:46.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.725+0000 7f995cd6c640 1 librados: calling monclient init 2024-03-29T10:55:46.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.725+0000 7f995cd6c640 10 monclient: init 2024-03-29T10:55:46.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:46.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:46.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 10 monclient(hunting): picked mon.noname-a con 0x7f99580a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:46.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 10 monclient(hunting): picked mon.noname-b con 0x7f995813f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:46.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 10 monclient(hunting): picked mon.noname-c con 0x7f99581457f0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:46.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580ac260 con 0x7f99580a9d70 2024-03-29T10:55:46.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9958137a20 con 0x7f995813f650 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99580ac4d0 con 0x7f99581457f0 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f9955d9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f995813f650 0x7f9958136c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:43324/0 (socket says 172.21.15.22:43324) 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f9955d9b640 1 -- 172.21.15.22:0/4116196695 learned_addr learned my addr 172.21.15.22:0/4116196695 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.726+0000 7f995cd6c640 10 monclient(hunting): authenticate will time out at 848.030999s 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 client.?.objecter ms_handle_connect 0x7f995813f650 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 client.?.objecter ms_handle_connect 0x7f99581457f0 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779067494 0 0) 0x7f9958137a20 con 0x7f995813f650 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:46.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): my global_id is 4294 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9938004eb0 con 0x7f995813f650 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2358762086 0 0) 0x7f99580ac4d0 con 0x7f99581457f0 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 10 monclient(hunting): my global_id is 4298 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.727+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9958137a20 con 0x7f99581457f0 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3875102165 0 0) 0x7f9958137a20 con 0x7f99581457f0 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99580ac4d0 con 0x7f99581457f0 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9950001ab0 con 0x7f99581457f0 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4025866774 0 0) 0x7f9938004eb0 con 0x7f995813f650 2024-03-29T10:55:46.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9958137a20 con 0x7f995813f650 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f994c004080 con 0x7f995813f650 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient(hunting): dump: 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:46.718 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1237787506 0 0) 0x7f99580ac4d0 con 0x7f99581457f0 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 >> v1:172.21.15.26:6789/0 conn(0x7f995813f650 legacy=0x7f9958136c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 >> v1:172.21.15.22:6789/0 conn(0x7f99580a9d70 legacy=0x7f99581425d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 monclient: found mon.c 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99580ab620 con 0x7f99581457f0 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient: _finish_auth 0 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 5 monclient: authenticate success, global_id 4298 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 10 monclient: _renew_subs 2024-03-29T10:55:46.719 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9958142ce0 con 0x7f99581457f0 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 10 monclient: _renew_subs 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f995813ff80 con 0x7f99581457f0 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f995cd6c640 1 librados: init done 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.728+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9950001fc0 con 0x7f99581457f0 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.729+0000 7f9946ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.729+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9950003b10 con 0x7f99581457f0 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.729+0000 7f9946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.729+0000 7f9946ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.729+0000 7f9946ffd640 10 monclient: dump: 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:46.720 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.729+0000 7f9954d99640 4 set_mon_vals no callback set 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.730+0000 7f995cd6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.730+0000 7f995cd6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.730+0000 7f995cd6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.730+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9958075f10 con 0x7f99581457f0 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.731+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f9950019ae0 con 0x7f99581457f0 2024-03-29T10:55:46.721 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.731+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f9950083ab0 con 0x7f99581457f0 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.731+0000 7f9946ffd640 10 client.4298.objecter ms_dispatch 0x7f9958000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.731+0000 7f9946ffd640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.731+0000 7f9946ffd640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.731+0000 7f9946ffd640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.732+0000 7f995659c640 10 monclient: get_auth_request con 0x7f993806de30 auth_method 0 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.732+0000 7f9946ffd640 10 client.4298.objecter ms_handle_connect 0x7f993806de30 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.739+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f995004f390 con 0x7f99581457f0 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.739+0000 7f9946ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:46.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:46.739+0000 7f9946ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:47.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.063+0000 7f995cd6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:47.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.063+0000 7f995cd6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:47.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.063+0000 7f995cd6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:47.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.063+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9958142e70 con 0x7f99581457f0 2024-03-29T10:55:47.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.065+0000 7f9946ffd640 1 -- 172.21.15.22:0/4116196695 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f9958142e70 con 0x7f99581457f0 2024-03-29T10:55:47.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.065+0000 7f9946ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:47.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.065+0000 7f9946ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:47.045 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:47.046 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 10 librados: watch_flush enter 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 10 librados: watch_flush exit 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 10 client.4298.objecter successfully canceled tick 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 >> v1:172.21.15.22:6816/380221644 conn(0x7f993806de30 legacy=0x7f99380702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 10 monclient: shutdown 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 >> v1:172.21.15.22:6790/0 conn(0x7f99581457f0 legacy=0x7f9958137310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 shutdown_connections 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 >> 172.21.15.22:0/4116196695 conn(0x7f9958066a00 msgr2=0x7f99580b4920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 shutdown_connections 2024-03-29T10:55:47.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 1 -- 172.21.15.22:0/4116196695 wait complete. 2024-03-29T10:55:47.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:47.068+0000 7f995cd6c640 1 librados: shutdown 2024-03-29T10:55:47.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:55:53.870 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:55:54.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.032+0000 7f6839d82640 10 monclient: get_monmap_and_config 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.032+0000 7f6839d82640 10 monclient: build_initial_monmap 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.032+0000 7f6839d82640 10 monclient: monmap: 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:55:54.032946+0000 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:55:54.032946+0000 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:55:54.013 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.033+0000 7f6839d82640 1 Processor -- start 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.033+0000 7f6839d82640 1 -- start start 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.033+0000 7f6839d82640 10 monclient: init 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 10 monclient(hunting): picked mon.noname-a con 0x7f68341422b0 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 10 monclient(hunting): picked mon.noname-c con 0x7f6834142690 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 10 monclient(hunting): picked mon.noname-b con 0x7f68340b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:54.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6833fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f68340b17d0 0x7f68340b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:36584/0 (socket says 172.21.15.22:36584) 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6833fff640 1 -- 172.21.15.22:0/3579647503 learned_addr learned my addr 172.21.15.22:0/3579647503 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6834142c80 con 0x7f68341422b0 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340aae40 con 0x7f6834142690 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f683413fbd0 con 0x7f68340b17d0 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.034+0000 7f6839d82640 10 monclient(hunting): authenticate will time out at 855.338594s 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276691754 0 0) 0x7f683413fbd0 con 0x7f68340b17d0 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 10 monclient(hunting): _negotiate 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:54.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:54.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 10 monclient(hunting): my global_id is 4301 2024-03-29T10:55:54.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f681c003520 con 0x7f68340b17d0 2024-03-29T10:55:54.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3543459631 0 0) 0x7f681c003520 con 0x7f68340b17d0 2024-03-29T10:55:54.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 >> v1:172.21.15.26:6789/0 conn(0x7f6834142690 legacy=0x7f68340b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:55:54.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.035+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 >> v1:172.21.15.22:6789/0 conn(0x7f68341422b0 legacy=0x7f683413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: _finish_hunting 0 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 1 monclient: found mon.noname-b 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6834150930 con 0x7f68340b17d0 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: _finish_auth 0 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: _check_auth_tickets 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f682c001a10 con 0x7f68340b17d0 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:55:54.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: dump: 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f6839d82640 5 monclient: authenticate success, global_id 4301 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f6839d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f682c001b70 con 0x7f68340b17d0 2024-03-29T10:55:54.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 1 -- 172.21.15.22:0/3579647503 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f682c001fc0 con 0x7f68340b17d0 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f68327fc640 10 monclient: dump: 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:54.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f6839d82640 10 monclient: get_monmap_and_config success 2024-03-29T10:55:54.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.037+0000 7f6839d82640 4 set_mon_vals no callback set 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.038+0000 7f6839d82640 10 monclient: shutdown 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.038+0000 7f6839d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.038+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 >> v1:172.21.15.22:6790/0 conn(0x7f68340b17d0 legacy=0x7f68340b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.038+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 shutdown_connections 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.038+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 >> 172.21.15.22:0/3579647503 conn(0x7f6834066a00 msgr2=0x7f6834066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.038+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 shutdown_connections 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.039+0000 7f6839d82640 1 -- 172.21.15.22:0/3579647503 wait complete. 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.040+0000 7f6839d82640 10 monclient: build_initial_monmap 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.040+0000 7f6839d82640 10 monclient: monmap: 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:55:54.022 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.040+0000 7f6839d82640 1 librados: starting msgr at 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.040+0000 7f6839d82640 1 librados: starting objecter 2024-03-29T10:55:54.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.041+0000 7f6839d82640 1 Processor -- start 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.041+0000 7f6839d82640 1 -- start start 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.041+0000 7f6839d82640 1 librados: setting wanted keys 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.041+0000 7f6839d82640 1 librados: calling monclient init 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.041+0000 7f6839d82640 10 monclient: init 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 10 monclient: _reopen_session rank -1 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 10 monclient(hunting): picked mon.noname-c con 0x7f68340b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 10 monclient(hunting): picked mon.noname-b con 0x7f68341422b0 addr v1:172.21.15.26:6789/0 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 10 monclient(hunting): picked mon.noname-a con 0x7f6834142690 addr v1:172.21.15.22:6789/0 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340ae110 con 0x7f6834142690 2024-03-29T10:55:54.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68340ae380 con 0x7f68341422b0 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6834130340 con 0x7f68340b17d0 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 10 monclient(hunting): _renew_subs 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f6839d82640 10 monclient(hunting): authenticate will time out at 855.346340s 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f68337fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f68340b17d0 0x7f68340abf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37142/0 (socket says 172.21.15.22:37142) 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.042+0000 7f68337fe640 1 -- 172.21.15.22:0/1621365962 learned_addr learned my addr 172.21.15.22:0/1621365962 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 10 client.?.objecter ms_handle_connect 0x7f68340b17d0 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:55:54.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135260002 0 0) 0x7f6834130340 con 0x7f68340b17d0 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 10 monclient(hunting): _negotiate 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 10 monclient(hunting): my global_id is 4304 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.043+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6820004eb0 con 0x7f68340b17d0 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4133072632 0 0) 0x7f6820004eb0 con 0x7f68340b17d0 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6834130340 con 0x7f68340b17d0 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6824004080 con 0x7f68340b17d0 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 10 monclient(hunting): dump: 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.026 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3982416524 0 0) 0x7f6834130340 con 0x7f68340b17d0 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 >> v1:172.21.15.26:6789/0 conn(0x7f68341422b0 legacy=0x7f683412fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:54.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.044+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 >> v1:172.21.15.22:6789/0 conn(0x7f6834142690 legacy=0x7f6834131ae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: _finish_hunting 0 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 1 monclient: found mon.c 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68341304d0 con 0x7f68340b17d0 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: _finish_auth 0 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: _check_auth_tickets 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 5 monclient: authenticate success, global_id 4304 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 10 monclient: _renew_subs 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68340ac7a0 con 0x7f68340b17d0 2024-03-29T10:55:54.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 10 monclient: _renew_subs 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68341321f0 con 0x7f68340b17d0 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6839d82640 1 librados: init done 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68240043a0 con 0x7f68340b17d0 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6824001320 con 0x7f68340b17d0 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f681bfff640 10 monclient: dump: 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.030 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.046+0000 7f6831ffb640 4 set_mon_vals no callback set 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.048+0000 7f6839d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.048+0000 7f6839d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.048+0000 7f6839d82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:54.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.048+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6814005080 con 0x7f68340b17d0 2024-03-29T10:55:54.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.049+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f6824001480 con 0x7f68340b17d0 2024-03-29T10:55:54.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.050+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f6824084af0 con 0x7f68340b17d0 2024-03-29T10:55:54.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.050+0000 7f681bfff640 10 client.4304.objecter ms_dispatch 0x7f6834000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:55:54.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.050+0000 7f681bfff640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:55:54.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.050+0000 7f681bfff640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:55:54.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.050+0000 7f681bfff640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-29T10:55:54.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.050+0000 7f6832ffd640 10 monclient: get_auth_request con 0x7f682006dd00 auth_method 0 2024-03-29T10:55:54.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.050+0000 7f681bfff640 10 client.4304.objecter ms_handle_connect 0x7f682006dd00 2024-03-29T10:55:54.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.056+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68240503d0 con 0x7f68340b17d0 2024-03-29T10:55:54.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.057+0000 7f681bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:55:54.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.057+0000 7f681bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:55:54.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.363+0000 7f6839d82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:54.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.363+0000 7f6839d82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:54.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.363+0000 7f6839d82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:55:54.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.363+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6814005330 con 0x7f68340b17d0 2024-03-29T10:55:54.345 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:55:54.345 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:55:54.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.364+0000 7f681bfff640 1 -- 172.21.15.22:0/1621365962 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f6824052300 con 0x7f68340b17d0 2024-03-29T10:55:54.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.364+0000 7f681bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:55:54.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.364+0000 7f681bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.367+0000 7f6839d82640 10 librados: watch_flush enter 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 10 librados: watch_flush exit 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 10 client.4304.objecter successfully canceled tick 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 >> v1:172.21.15.22:6816/380221644 conn(0x7f682006dd00 legacy=0x7f68200701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 10 monclient: shutdown 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:55:54.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 >> v1:172.21.15.22:6790/0 conn(0x7f68340b17d0 legacy=0x7f68340abf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:55:54.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 shutdown_connections 2024-03-29T10:55:54.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 >> 172.21.15.22:0/1621365962 conn(0x7f6834066a00 msgr2=0x7f683414f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:55:54.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 shutdown_connections 2024-03-29T10:55:54.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.368+0000 7f6839d82640 1 -- 172.21.15.22:0/1621365962 wait complete. 2024-03-29T10:55:54.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:55:54.369+0000 7f6839d82640 1 librados: shutdown 2024-03-29T10:55:54.364 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:01.171 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.325+0000 7fee5e855640 10 monclient: get_monmap_and_config 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.325+0000 7fee5e855640 10 monclient: build_initial_monmap 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.325+0000 7fee5e855640 10 monclient: monmap: 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:01.326025+0000 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:01.326025+0000 2024-03-29T10:56:01.306 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:01.307 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:01.307 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:01.307 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:01.307 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:01.307 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:01.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.326+0000 7fee5e855640 1 Processor -- start 2024-03-29T10:56:01.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.326+0000 7fee5e855640 1 -- start start 2024-03-29T10:56:01.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.326+0000 7fee5e855640 10 monclient: init 2024-03-29T10:56:01.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:01.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:01.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 10 monclient(hunting): picked mon.noname-b con 0x7fee580b4920 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:01.310 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 10 monclient(hunting): picked mon.noname-a con 0x7fee580b1790 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 10 monclient(hunting): picked mon.noname-c con 0x7fee58146790 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee580b4070 con 0x7fee580b1790 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5806fb80 con 0x7fee58146790 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee580b42c0 con 0x7fee580b4920 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee57fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fee580b4920 0x7fee5806e510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53254/0 (socket says 172.21.15.22:53254) 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee5e855640 10 monclient(hunting): authenticate will time out at 862.631429s 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee57fff640 1 -- 172.21.15.22:0/923660817 learned_addr learned my addr 172.21.15.22:0/923660817 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.327+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382525104 0 0) 0x7fee580b42c0 con 0x7fee580b4920 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:01.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): my global_id is 4307 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee40003520 con 0x7fee580b4920 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702412726 0 0) 0x7fee5806fb80 con 0x7fee58146790 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 10 monclient(hunting): my global_id is 4297 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee580b42c0 con 0x7fee58146790 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1941710997 0 0) 0x7fee580b42c0 con 0x7fee58146790 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.328+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 >> v1:172.21.15.22:6790/0 conn(0x7fee580b4920 legacy=0x7fee5806e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:01.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.329+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 >> v1:172.21.15.22:6789/0 conn(0x7fee580b1790 legacy=0x7fee580b1bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 1 monclient: found mon.noname-c 2024-03-29T10:56:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:56:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee580b22c0 con 0x7fee58146790 2024-03-29T10:56:01.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fee500019f0 con 0x7fee58146790 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee5e855640 5 monclient: authenticate success, global_id 4297 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee5e855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: dump: 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.316 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee50001c10 con 0x7fee58146790 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee5e855640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee56ffd640 1 -- 172.21.15.22:0/923660817 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fee50001fa0 con 0x7fee58146790 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee5e855640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.331+0000 7fee5e855640 4 set_mon_vals no callback set 2024-03-29T10:56:01.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee56ffd640 10 monclient: dump: 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:01.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee5e855640 10 monclient: shutdown 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee5e855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee5e855640 1 -- 172.21.15.22:0/923660817 >> v1:172.21.15.26:6789/0 conn(0x7fee58146790 legacy=0x7fee58146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee5e855640 1 -- 172.21.15.22:0/923660817 shutdown_connections 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee5e855640 1 -- 172.21.15.22:0/923660817 >> 172.21.15.22:0/923660817 conn(0x7fee58066a00 msgr2=0x7fee58066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.332+0000 7fee5e855640 1 -- 172.21.15.22:0/923660817 shutdown_connections 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.333+0000 7fee5e855640 1 -- 172.21.15.22:0/923660817 wait complete. 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.334+0000 7fee5e855640 10 monclient: build_initial_monmap 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.334+0000 7fee5e855640 10 monclient: monmap: 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:01.319 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.334+0000 7fee5e855640 1 librados: starting msgr at 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.334+0000 7fee5e855640 1 librados: starting objecter 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.334+0000 7fee5e855640 1 Processor -- start 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.335+0000 7fee5e855640 1 -- start start 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.335+0000 7fee5e855640 1 librados: setting wanted keys 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.335+0000 7fee5e855640 1 librados: calling monclient init 2024-03-29T10:56:01.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.335+0000 7fee5e855640 10 monclient: init 2024-03-29T10:56:01.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.335+0000 7fee5e855640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:01.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee5e855640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:01.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee5e855640 10 monclient(hunting): picked mon.noname-b con 0x7fee580b1790 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:01.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee5e855640 10 monclient(hunting): picked mon.noname-c con 0x7fee580b4920 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee577fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fee580b4920 0x7fee58137ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53650/0 (socket says 172.21.15.22:53650) 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee577fe640 1 -- 172.21.15.22:0/1539750029 learned_addr learned my addr 172.21.15.22:0/1539750029 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee54ff9640 10 client.?.objecter ms_handle_connect 0x7fee580b4920 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee5e855640 10 monclient(hunting): picked mon.noname-a con 0x7fee58146790 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee58139780 con 0x7fee58146790 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.336+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee581399d0 con 0x7fee580b1790 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 client.?.objecter ms_handle_connect 0x7fee580b1790 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5813bbf0 con 0x7fee580b4920 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee5e855640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee5e855640 10 monclient(hunting): authenticate will time out at 862.641138s 2024-03-29T10:56:01.324 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683178465 0 0) 0x7fee5813bbf0 con 0x7fee580b4920 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): my global_id is 4310 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee44004eb0 con 0x7fee580b4920 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936183108 0 0) 0x7fee581399d0 con 0x7fee580b1790 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): my global_id is 4300 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5813bbf0 con 0x7fee580b1790 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113051322 0 0) 0x7fee44004eb0 con 0x7fee580b4920 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee581399d0 con 0x7fee580b4920 2024-03-29T10:56:01.325 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fee4c001960 con 0x7fee580b4920 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.337+0000 7fee54ff9640 10 monclient(hunting): dump: 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:01.326 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.338+0000 7fee54ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.338+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 29643940 0 0) 0x7fee581399d0 con 0x7fee580b4920 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.338+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 >> v1:172.21.15.26:6789/0 conn(0x7fee580b1790 legacy=0x7fee5813a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.338+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 >> v1:172.21.15.22:6789/0 conn(0x7fee58146790 legacy=0x7fee581385f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 1 monclient: found mon.c 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:01.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee58132710 con 0x7fee580b4920 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 10 monclient: _finish_auth 0 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee54ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 5 monclient: authenticate success, global_id 4310 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 10 monclient: _renew_subs 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee5813c110 con 0x7fee580b4920 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 10 monclient: _renew_subs 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee581e7370 con 0x7fee580b4920 2024-03-29T10:56:01.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.339+0000 7fee5e855640 1 librados: init done 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee4c001ac0 con 0x7fee580b4920 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee54ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fee4c0038a0 con 0x7fee580b4920 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee54ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee54ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee54ff9640 10 monclient: dump: 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:01.329 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee5e855640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee5e855640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee5e855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.340+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5813be30 con 0x7fee580b4920 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.341+0000 7fee567fc640 4 set_mon_vals no callback set 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.341+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fee4c003a00 con 0x7fee580b4920 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.342+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fee4c084a30 con 0x7fee580b4920 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.342+0000 7fee54ff9640 10 client.4310.objecter ms_dispatch 0x7fee58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.342+0000 7fee54ff9640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:01.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.342+0000 7fee54ff9640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:01.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.342+0000 7fee54ff9640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-29T10:56:01.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.342+0000 7fee57fff640 10 monclient: get_auth_request con 0x7fee4406dd90 auth_method 0 2024-03-29T10:56:01.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.343+0000 7fee54ff9640 10 client.4310.objecter ms_handle_connect 0x7fee4406dd90 2024-03-29T10:56:01.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.349+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee5813be30 con 0x7fee580b4920 2024-03-29T10:56:01.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.349+0000 7fee54ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:01.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.349+0000 7fee54ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:01.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.670+0000 7fee5e855640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:01.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.670+0000 7fee5e855640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:01.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.670+0000 7fee5e855640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:01.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.670+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee5813be30 con 0x7fee580b4920 2024-03-29T10:56:01.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.672+0000 7fee54ff9640 1 -- 172.21.15.22:0/1539750029 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fee5813be30 con 0x7fee580b4920 2024-03-29T10:56:01.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.672+0000 7fee54ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:01.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.672+0000 7fee54ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:01.654 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:01.655 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 10 librados: watch_flush enter 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 10 librados: watch_flush exit 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 10 client.4310.objecter successfully canceled tick 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 >> v1:172.21.15.22:6816/380221644 conn(0x7fee4406dd90 legacy=0x7fee44070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 10 monclient: shutdown 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:01.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 >> v1:172.21.15.22:6790/0 conn(0x7fee580b4920 legacy=0x7fee58137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:01.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 shutdown_connections 2024-03-29T10:56:01.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.677+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 >> 172.21.15.22:0/1539750029 conn(0x7fee58066a00 msgr2=0x7fee5806e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:01.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.678+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 shutdown_connections 2024-03-29T10:56:01.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.678+0000 7fee5e855640 1 -- 172.21.15.22:0/1539750029 wait complete. 2024-03-29T10:56:01.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:01.678+0000 7fee5e855640 1 librados: shutdown 2024-03-29T10:56:01.674 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:08.481 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:08.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.652+0000 7f53d6d77640 10 monclient: get_monmap_and_config 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.652+0000 7f53d6d77640 10 monclient: build_initial_monmap 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.652+0000 7f53d6d77640 10 monclient: monmap: 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:08.654522+0000 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:08.654522+0000 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.653+0000 7f53d6d77640 1 Processor -- start 2024-03-29T10:56:08.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 1 -- start start 2024-03-29T10:56:08.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient: init 2024-03-29T10:56:08.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:08.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:56:08.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient(hunting): picked mon.noname-a con 0x7f53d00a9ac0 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:08.638 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient(hunting): picked mon.noname-c con 0x7f53d013ee90 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient(hunting): picked mon.noname-b con 0x7f53d00af900 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d00afd10 con 0x7f53d00a9ac0 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d00af6f0 con 0x7f53d013ee90 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d00aac80 con 0x7f53d00af900 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.654+0000 7f53d6d77640 10 monclient(hunting): authenticate will time out at 869.960067s 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.655+0000 7f53d5314640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f53d00af900 0x7f53d0149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34058/0 (socket says 172.21.15.22:34058) 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.655+0000 7f53d5314640 1 -- 172.21.15.22:0/3215820785 learned_addr learned my addr 172.21.15.22:0/3215820785 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76183966 0 0) 0x7f53d00aac80 con 0x7f53d00af900 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:08.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:08.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:08.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient(hunting): my global_id is 4313 2024-03-29T10:56:08.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b8003520 con 0x7f53d00af900 2024-03-29T10:56:08.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 595276143 0 0) 0x7f53b8003520 con 0x7f53d00af900 2024-03-29T10:56:08.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 >> v1:172.21.15.26:6789/0 conn(0x7f53d013ee90 legacy=0x7f53d013b5f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:56:08.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 >> v1:172.21.15.22:6789/0 conn(0x7f53d00a9ac0 legacy=0x7f53d00adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 monclient: found mon.noname-b 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d00aaf60 con 0x7f53d00af900 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53c8001a10 con 0x7f53d00af900 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.658+0000 7f53cf7fe640 10 monclient: dump: 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:08.641 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:08.642 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.659+0000 7f53d6d77640 5 monclient: authenticate success, global_id 4313 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.659+0000 7f53d6d77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.659+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c8001be0 con 0x7f53d00af900 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.659+0000 7f53cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.659+0000 7f53cf7fe640 1 -- 172.21.15.22:0/3215820785 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53c8002030 con 0x7f53d00af900 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.660+0000 7f53d6d77640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.660+0000 7f53d6d77640 4 set_mon_vals no callback set 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.660+0000 7f53cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.660+0000 7f53cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.660+0000 7f53cf7fe640 10 monclient: dump: 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:08.643 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.662+0000 7f53d6d77640 10 monclient: shutdown 2024-03-29T10:56:08.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.662+0000 7f53d6d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:08.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.662+0000 7f53d6d77640 1 -- 172.21.15.22:0/3215820785 >> v1:172.21.15.22:6790/0 conn(0x7f53d00af900 legacy=0x7f53d0149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:08.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.662+0000 7f53d6d77640 1 -- 172.21.15.22:0/3215820785 shutdown_connections 2024-03-29T10:56:08.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.662+0000 7f53d6d77640 1 -- 172.21.15.22:0/3215820785 >> 172.21.15.22:0/3215820785 conn(0x7f53d00b35e0 msgr2=0x7f53d00b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:08.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.664+0000 7f53d6d77640 1 -- 172.21.15.22:0/3215820785 shutdown_connections 2024-03-29T10:56:08.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.664+0000 7f53d6d77640 1 -- 172.21.15.22:0/3215820785 wait complete. 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.665+0000 7f53d6d77640 10 monclient: build_initial_monmap 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.665+0000 7f53d6d77640 10 monclient: monmap: 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:08.661 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.666+0000 7f53d6d77640 1 librados: starting msgr at 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.666+0000 7f53d6d77640 1 librados: starting objecter 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.666+0000 7f53d6d77640 1 Processor -- start 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.666+0000 7f53d6d77640 1 -- start start 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.666+0000 7f53d6d77640 1 librados: setting wanted keys 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.666+0000 7f53d6d77640 1 librados: calling monclient init 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.666+0000 7f53d6d77640 10 monclient: init 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 10 monclient(hunting): picked mon.noname-b con 0x7f53d00a9ac0 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 10 monclient(hunting): picked mon.noname-a con 0x7f53d00af900 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 10 monclient(hunting): picked mon.noname-c con 0x7f53d013ee90 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:08.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d0140820 con 0x7f53d00af900 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d0140a90 con 0x7f53d00a9ac0 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d0140d70 con 0x7f53d013ee90 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.667+0000 7f53d6d77640 10 monclient(hunting): authenticate will time out at 869.972516s 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cffff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f53d00af900 0x7f53d0141190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48587/0 (socket says 172.21.15.22:48587) 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cffff640 1 -- 172.21.15.22:0/2372096938 learned_addr learned my addr 172.21.15.22:0/2372096938 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 client.?.objecter ms_handle_connect 0x7f53d00af900 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 client.?.objecter ms_handle_connect 0x7f53d013ee90 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230276793 0 0) 0x7f53d0140820 con 0x7f53d00af900 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:08.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): my global_id is 4293 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53bc004db0 con 0x7f53d00af900 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543676884 0 0) 0x7f53d0140d70 con 0x7f53d013ee90 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): my global_id is 4316 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d0140820 con 0x7f53d013ee90 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2818572811 0 0) 0x7f53d0140820 con 0x7f53d013ee90 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53d0140d70 con 0x7f53d013ee90 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53c80013b0 con 0x7f53d013ee90 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.671+0000 7f53cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714387914 0 0) 0x7f53bc004db0 con 0x7f53d00af900 2024-03-29T10:56:08.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53d0140820 con 0x7f53d00af900 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53c4001870 con 0x7f53d00af900 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:08.666 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2622531165 0 0) 0x7f53d0140820 con 0x7f53d00af900 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 >> v1:172.21.15.22:6790/0 conn(0x7f53d013ee90 legacy=0x7f53d01ef820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.672+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 >> v1:172.21.15.26:6789/0 conn(0x7f53d00a9ac0 legacy=0x7f53d0138aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 1 monclient: found mon.a 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d01391b0 con 0x7f53d00af900 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 5 monclient: authenticate success, global_id 4293 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 10 monclient: _renew_subs 2024-03-29T10:56:08.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d0139ad0 con 0x7f53d00af900 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 10 monclient: _renew_subs 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d013a040 con 0x7f53d00af900 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.674+0000 7f53d6d77640 1 librados: init done 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.675+0000 7f53d6d77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.675+0000 7f53d6d77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.675+0000 7f53d6d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.675+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d0075f10 con 0x7f53d00af900 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.676+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53c4001d80 con 0x7f53d00af900 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.676+0000 7f53cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.676+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53c4004aa0 con 0x7f53d00af900 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.676+0000 7f53cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:08.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.676+0000 7f53cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.676+0000 7f53cd7fa640 10 monclient: dump: 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:08.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.676+0000 7f53ceffd640 4 set_mon_vals no callback set 2024-03-29T10:56:08.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.679+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f53c401aa70 con 0x7f53d00af900 2024-03-29T10:56:08.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.680+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f53c4084a00 con 0x7f53d00af900 2024-03-29T10:56:08.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.680+0000 7f53cd7fa640 10 client.4293.objecter ms_dispatch 0x7f53d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:08.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.680+0000 7f53cd7fa640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:08.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.680+0000 7f53cd7fa640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:08.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.680+0000 7f53cd7fa640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-29T10:56:08.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.682+0000 7f53d4b13640 10 monclient: get_auth_request con 0x7f53bc0766a0 auth_method 0 2024-03-29T10:56:08.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.682+0000 7f53cd7fa640 10 client.4293.objecter ms_handle_connect 0x7f53bc0766a0 2024-03-29T10:56:08.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.685+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53c4050110 con 0x7f53d00af900 2024-03-29T10:56:08.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.685+0000 7f53cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:08.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:08.685+0000 7f53cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:09.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.028+0000 7f53d6d77640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:09.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.028+0000 7f53d6d77640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:09.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.028+0000 7f53d6d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:09.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.028+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53d0139c60 con 0x7f53d00af900 2024-03-29T10:56:09.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.029+0000 7f53cd7fa640 1 -- 172.21.15.22:0/2372096938 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f53c4052210 con 0x7f53d00af900 2024-03-29T10:56:09.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.029+0000 7f53cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:09.011 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.029+0000 7f53cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:09.012 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:09.012 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:09.014 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.033+0000 7f53d6d77640 10 librados: watch_flush enter 2024-03-29T10:56:09.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.033+0000 7f53d6d77640 10 librados: watch_flush exit 2024-03-29T10:56:09.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.033+0000 7f53d6d77640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-29T10:56:09.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.033+0000 7f53d6d77640 10 client.4293.objecter successfully canceled tick 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.033+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 >> v1:172.21.15.22:6816/380221644 conn(0x7f53bc0766a0 legacy=0x7f53bc078b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.033+0000 7f53d6d77640 10 monclient: shutdown 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.034+0000 7f53d6d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.034+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 >> v1:172.21.15.22:6789/0 conn(0x7f53d00af900 legacy=0x7f53d0141190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.034+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 shutdown_connections 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.034+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 >> 172.21.15.22:0/2372096938 conn(0x7f53d00b35e0 msgr2=0x7f53d013b190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.034+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 shutdown_connections 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.034+0000 7f53d6d77640 1 -- 172.21.15.22:0/2372096938 wait complete. 2024-03-29T10:56:09.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:09.034+0000 7f53d6d77640 1 librados: shutdown 2024-03-29T10:56:09.033 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:15.839 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:15.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.009+0000 7f5c62a3f640 10 monclient: get_monmap_and_config 2024-03-29T10:56:15.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.009+0000 7f5c62a3f640 10 monclient: build_initial_monmap 2024-03-29T10:56:15.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.009+0000 7f5c62a3f640 10 monclient: monmap: 2024-03-29T10:56:15.991 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:15.991 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:16.011544+0000 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:16.011544+0000 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:15.992 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:15.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.011+0000 7f5c62a3f640 1 Processor -- start 2024-03-29T10:56:15.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.012+0000 7f5c62a3f640 1 -- start start 2024-03-29T10:56:15.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.012+0000 7f5c62a3f640 10 monclient: init 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5c0b2800 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 10 monclient(hunting): picked mon.noname-a con 0x7f5c5c0b3220 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5c1439c0 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c0b3840 con 0x7f5c5c0b3220 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c06f230 con 0x7f5c5c0b2800 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c06ef50 con 0x7f5c5c1439c0 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:15.996 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.013+0000 7f5c62a3f640 10 monclient(hunting): authenticate will time out at 877.318254s 2024-03-29T10:56:15.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c6223e640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5c5c1439c0 0x7f5c5c151c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58383/0 (socket says 172.21.15.22:58383) 2024-03-29T10:56:15.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c6223e640 1 -- 172.21.15.22:0/2349218538 learned_addr learned my addr 172.21.15.22:0/2349218538 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3827861530 0 0) 0x7f5c5c06ef50 con 0x7f5c5c1439c0 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c60a3b640 10 monclient(hunting): _negotiate 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c60a3b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c60a3b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c60a3b640 10 monclient(hunting): my global_id is 4319 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.016+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c48003520 con 0x7f5c5c1439c0 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.017+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1062935658 0 0) 0x7f5c48003520 con 0x7f5c5c1439c0 2024-03-29T10:56:15.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.017+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 >> v1:172.21.15.26:6789/0 conn(0x7f5c5c0b2800 legacy=0x7f5c5c1432b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:56:16.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 >> v1:172.21.15.22:6789/0 conn(0x7f5c5c0b3220 legacy=0x7f5c5c146710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 10 monclient: _finish_hunting 0 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 1 monclient: found mon.noname-b 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c071d30 con 0x7f5c5c1439c0 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 10 monclient: _finish_auth 0 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 10 monclient: _check_auth_tickets 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c62a3f640 5 monclient: authenticate success, global_id 4319 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c62a3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5c54001a10 con 0x7f5c5c1439c0 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:16.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c60a3b640 10 monclient: dump: 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.020+0000 7f5c62a3f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c54001d20 con 0x7f5c5c1439c0 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c60a3b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:16.005 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c62a3f640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c62a3f640 4 set_mon_vals no callback set 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c60a3b640 1 -- 172.21.15.22:0/2349218538 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5c540020b0 con 0x7f5c5c1439c0 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c60a3b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c60a3b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.021+0000 7f5c60a3b640 10 monclient: dump: 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:16.006 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:16.007 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:16.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.022+0000 7f5c62a3f640 10 monclient: shutdown 2024-03-29T10:56:16.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.022+0000 7f5c62a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:16.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.022+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2349218538 >> v1:172.21.15.22:6790/0 conn(0x7f5c5c1439c0 legacy=0x7f5c5c151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:16.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.022+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2349218538 shutdown_connections 2024-03-29T10:56:16.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.022+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2349218538 >> 172.21.15.22:0/2349218538 conn(0x7f5c5c066a00 msgr2=0x7f5c5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:16.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.026+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2349218538 shutdown_connections 2024-03-29T10:56:16.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.026+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2349218538 wait complete. 2024-03-29T10:56:16.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.027+0000 7f5c62a3f640 10 monclient: build_initial_monmap 2024-03-29T10:56:16.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.027+0000 7f5c62a3f640 10 monclient: monmap: 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:16.009 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.027+0000 7f5c62a3f640 1 librados: starting msgr at 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.027+0000 7f5c62a3f640 1 librados: starting objecter 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 1 Processor -- start 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 1 -- start start 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 1 librados: setting wanted keys 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 1 librados: calling monclient init 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient: init 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:16.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient(hunting): picked mon.noname-c con 0x7f5c5c0b2800 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient(hunting): picked mon.noname-b con 0x7f5c5c0b3220 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient(hunting): picked mon.noname-a con 0x7f5c5c1439c0 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c1201d0 con 0x7f5c5c1439c0 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c121300 con 0x7f5c5c0b3220 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c5c13ea40 con 0x7f5c5c0b2800 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:16.012 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.028+0000 7f5c62a3f640 10 monclient(hunting): authenticate will time out at 877.334027s 2024-03-29T10:56:16.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.031+0000 7f5c61a3d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5c5c0b2800 0x7f5c5c11fac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58453/0 (socket says 172.21.15.22:58453) 2024-03-29T10:56:16.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.031+0000 7f5c61a3d640 1 -- 172.21.15.22:0/2081799368 learned_addr learned my addr 172.21.15.22:0/2081799368 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:16.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.031+0000 7f5c527fc640 10 client.?.objecter ms_handle_connect 0x7f5c5c0b2800 2024-03-29T10:56:16.013 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.031+0000 7f5c527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:16.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.035+0000 7f5c527fc640 10 client.?.objecter ms_handle_connect 0x7f5c5c0b3220 2024-03-29T10:56:16.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.035+0000 7f5c527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:16.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.035+0000 7f5c527fc640 10 client.?.objecter ms_handle_connect 0x7f5c5c1439c0 2024-03-29T10:56:16.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.035+0000 7f5c527fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:16.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.036+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2095248743 0 0) 0x7f5c5c121300 con 0x7f5c5c0b3220 2024-03-29T10:56:16.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.036+0000 7f5c527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:16.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.036+0000 7f5c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:16.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.036+0000 7f5c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:16.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.036+0000 7f5c527fc640 10 monclient(hunting): my global_id is 4303 2024-03-29T10:56:16.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.036+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c40004eb0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.037+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 997209628 0 0) 0x7f5c5c13ea40 con 0x7f5c5c0b2800 2024-03-29T10:56:16.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.037+0000 7f5c527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:16.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.037+0000 7f5c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:16.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.037+0000 7f5c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:16.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.037+0000 7f5c527fc640 10 monclient(hunting): my global_id is 4322 2024-03-29T10:56:16.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.037+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c5c121300 con 0x7f5c5c0b2800 2024-03-29T10:56:16.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.038+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3894557090 0 0) 0x7f5c5c1201d0 con 0x7f5c5c1439c0 2024-03-29T10:56:16.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.038+0000 7f5c527fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.038+0000 7f5c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.038+0000 7f5c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.038+0000 7f5c527fc640 10 monclient(hunting): my global_id is 4299 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.038+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c5c13ea40 con 0x7f5c5c1439c0 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1777131843 0 0) 0x7f5c40004eb0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5c1201d0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5c4c0018b0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 10 monclient(hunting): dump: 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:16.024 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1773795388 0 0) 0x7f5c5c13ea40 con 0x7f5c5c1439c0 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c40004eb0 con 0x7f5c5c1439c0 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5c540019d0 con 0x7f5c5c1439c0 2024-03-29T10:56:16.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.039+0000 7f5c527fc640 10 monclient(hunting): dump: 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:16.026 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1844987204 0 0) 0x7f5c5c121300 con 0x7f5c5c0b2800 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c5c13ea40 con 0x7f5c5c0b2800 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5c580017f0 con 0x7f5c5c0b2800 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 10 monclient(hunting): dump: 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:16.027 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2397764865 0 0) 0x7f5c5c1201d0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 >> v1:172.21.15.22:6790/0 conn(0x7f5c5c0b2800 legacy=0x7f5c5c11fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 >> v1:172.21.15.22:6789/0 conn(0x7f5c5c1439c0 legacy=0x7f5c5c1db6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 monclient: found mon.b 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c14a160 con 0x7f5c5c0b3220 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c62a3f640 5 monclient: authenticate success, global_id 4303 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c62a3f640 10 monclient: _renew_subs 2024-03-29T10:56:16.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c62a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:16.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.040+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c5c14a3b0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c62a3f640 10 client.4303.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:16.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c62a3f640 10 monclient: _renew_subs 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c62a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5c13e620 con 0x7f5c5c0b3220 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c62a3f640 1 librados: init done 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c4c0035f0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5c4c003900 con 0x7f5c5c0b3220 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.041+0000 7f5c527fc640 10 monclient: dump: 2024-03-29T10:56:16.031 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.042+0000 7f5c53fff640 4 set_mon_vals no callback set 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.043+0000 7f5c62a3f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.043+0000 7f5c62a3f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.043+0000 7f5c62a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.043+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c44005080 con 0x7f5c5c0b3220 2024-03-29T10:56:16.032 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.046+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f5c4c0198d0 con 0x7f5c5c0b3220 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.046+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f5c4c083720 con 0x7f5c5c0b3220 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.046+0000 7f5c527fc640 10 client.4303.objecter ms_dispatch 0x7f5c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.046+0000 7f5c527fc640 3 client.4303.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.046+0000 7f5c527fc640 3 client.4303.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.046+0000 7f5c527fc640 20 client.4303.objecter dump_active .. 0 homeless 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.048+0000 7f5c61a3d640 10 monclient: get_auth_request con 0x7f5c4006e120 auth_method 0 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.050+0000 7f5c527fc640 10 client.4303.objecter ms_handle_connect 0x7f5c4006e120 2024-03-29T10:56:16.033 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.051+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5c4c04ee30 con 0x7f5c5c0b3220 2024-03-29T10:56:16.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.051+0000 7f5c527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:16.034 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.051+0000 7f5c527fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:16.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.386+0000 7f5c62a3f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:16.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.387+0000 7f5c62a3f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:16.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.387+0000 7f5c62a3f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:16.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.387+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c44005330 con 0x7f5c5c0b3220 2024-03-29T10:56:16.372 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:16.372 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:16.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.390+0000 7f5c527fc640 1 -- 172.21.15.22:0/2081799368 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f5c4c050f30 con 0x7f5c5c0b3220 2024-03-29T10:56:16.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.390+0000 7f5c527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:16.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.390+0000 7f5c527fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:16.375 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.393+0000 7f5c62a3f640 10 librados: watch_flush enter 2024-03-29T10:56:16.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 10 librados: watch_flush exit 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 20 client.4303.objecter shutdown clearing up homeless session... 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 10 client.4303.objecter successfully canceled tick 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 >> v1:172.21.15.22:6816/380221644 conn(0x7f5c4006e120 legacy=0x7f5c400705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 10 monclient: shutdown 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 >> v1:172.21.15.26:6789/0 conn(0x7f5c5c0b3220 legacy=0x7f5c5c13dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 shutdown_connections 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.397+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 >> 172.21.15.22:0/2081799368 conn(0x7f5c5c066a00 msgr2=0x7f5c5c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.398+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 shutdown_connections 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.398+0000 7f5c62a3f640 1 -- 172.21.15.22:0/2081799368 wait complete. 2024-03-29T10:56:16.380 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:16.398+0000 7f5c62a3f640 1 librados: shutdown 2024-03-29T10:56:16.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:23.204 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:23.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.372+0000 7f571489a640 10 monclient: get_monmap_and_config 2024-03-29T10:56:23.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.372+0000 7f571489a640 10 monclient: build_initial_monmap 2024-03-29T10:56:23.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.372+0000 7f571489a640 10 monclient: monmap: 2024-03-29T10:56:23.353 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:23.353 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:23.353 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:23.373361+0000 2024-03-29T10:56:23.353 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:23.373361+0000 2024-03-29T10:56:23.354 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:23.354 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:23.354 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:23.354 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:23.354 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:23.354 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:23.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.373+0000 7f571489a640 1 Processor -- start 2024-03-29T10:56:23.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.373+0000 7f571489a640 1 -- start start 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.373+0000 7f571489a640 10 monclient: init 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 10 monclient(hunting): picked mon.noname-a con 0x7f57100b0770 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 10 monclient(hunting): picked mon.noname-b con 0x7f571014a700 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 10 monclient(hunting): picked mon.noname-c con 0x7f57100ac5a0 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57100aba70 con 0x7f57100b0770 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57100aecb0 con 0x7f57100ac5a0 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57100af150 con 0x7f571014a700 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:23.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.374+0000 7f571489a640 10 monclient(hunting): authenticate will time out at 884.678782s 2024-03-29T10:56:23.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.378+0000 7f570dd9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f571014a700 0x7f57100abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56363/0 (socket says 172.21.15.22:56363) 2024-03-29T10:56:23.359 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.378+0000 7f570dd9b640 1 -- 172.21.15.22:0/3794412066 learned_addr learned my addr 172.21.15.22:0/3794412066 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:23.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.379+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547641743 0 0) 0x7f57100af150 con 0x7f571014a700 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): my global_id is 4325 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f8003520 con 0x7f571014a700 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 965677392 0 0) 0x7f57100aba70 con 0x7f57100b0770 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): _negotiate 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient(hunting): my global_id is 4302 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57100af150 con 0x7f57100b0770 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3373084151 0 0) 0x7f56f8003520 con 0x7f571014a700 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 >> v1:172.21.15.26:6789/0 conn(0x7f57100ac5a0 legacy=0x7f5710151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:23.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 >> v1:172.21.15.22:6789/0 conn(0x7f57100b0770 legacy=0x7f5710149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient: _finish_hunting 0 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 monclient: found mon.noname-b 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57100ac9d0 con 0x7f571014a700 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient: _finish_auth 0 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient: _check_auth_tickets 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5704001690 con 0x7f571014a700 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f570d59a640 10 monclient: dump: 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:23.364 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f571489a640 5 monclient: authenticate success, global_id 4325 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.380+0000 7f571489a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57040017f0 con 0x7f571014a700 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f570d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f570d59a640 1 -- 172.21.15.22:0/3794412066 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5704001c00 con 0x7f571014a700 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f570d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f570d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:23.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f570d59a640 10 monclient: dump: 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f571489a640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.381+0000 7f571489a640 4 set_mon_vals no callback set 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.382+0000 7f571489a640 10 monclient: shutdown 2024-03-29T10:56:23.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.382+0000 7f571489a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:23.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.382+0000 7f571489a640 1 -- 172.21.15.22:0/3794412066 >> v1:172.21.15.22:6790/0 conn(0x7f571014a700 legacy=0x7f57100abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:23.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.382+0000 7f571489a640 1 -- 172.21.15.22:0/3794412066 shutdown_connections 2024-03-29T10:56:23.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.382+0000 7f571489a640 1 -- 172.21.15.22:0/3794412066 >> 172.21.15.22:0/3794412066 conn(0x7f571006b660 msgr2=0x7f571006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:23.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.382+0000 7f571489a640 1 -- 172.21.15.22:0/3794412066 shutdown_connections 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.382+0000 7f571489a640 1 -- 172.21.15.22:0/3794412066 wait complete. 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.383+0000 7f571489a640 10 monclient: build_initial_monmap 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.383+0000 7f571489a640 10 monclient: monmap: 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:23.369 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.384+0000 7f571489a640 1 librados: starting msgr at 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.384+0000 7f571489a640 1 librados: starting objecter 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.384+0000 7f571489a640 1 Processor -- start 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.384+0000 7f571489a640 1 -- start start 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.384+0000 7f571489a640 1 librados: setting wanted keys 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.384+0000 7f571489a640 1 librados: calling monclient init 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.384+0000 7f571489a640 10 monclient: init 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 10 monclient(hunting): picked mon.noname-c con 0x7f57100ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 10 monclient(hunting): picked mon.noname-b con 0x7f57100b0770 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:23.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 10 monclient(hunting): picked mon.noname-a con 0x7f571014a700 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:23.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571011f3b0 con 0x7f571014a700 2024-03-29T10:56:23.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571011e170 con 0x7f57100b0770 2024-03-29T10:56:23.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f571011dc50 con 0x7f57100ac5a0 2024-03-29T10:56:23.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:23.371 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.385+0000 7f571489a640 10 monclient(hunting): authenticate will time out at 884.689843s 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.391+0000 7f570dd9b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f57100b0770 0x7f5710147e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:50331/0 (socket says 172.21.15.22:50331) 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.391+0000 7f570dd9b640 1 -- 172.21.15.22:0/2870266193 learned_addr learned my addr 172.21.15.22:0/2870266193 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 client.?.objecter ms_handle_connect 0x7f57100b0770 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 client.?.objecter ms_handle_connect 0x7f57100ac5a0 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802540538 0 0) 0x7f571011e170 con 0x7f57100b0770 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): my global_id is 4306 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56f0004eb0 con 0x7f57100b0770 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910513546 0 0) 0x7f571011dc50 con 0x7f57100ac5a0 2024-03-29T10:56:23.376 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.392+0000 7f56feffd640 10 monclient(hunting): my global_id is 4328 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f571011e170 con 0x7f57100ac5a0 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2755895997 0 0) 0x7f56f0004eb0 con 0x7f57100b0770 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f571011dc50 con 0x7f57100b0770 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5704001c00 con 0x7f57100b0770 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 10 monclient(hunting): dump: 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.377 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2205790702 0 0) 0x7f571011dc50 con 0x7f57100b0770 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 >> v1:172.21.15.22:6790/0 conn(0x7f57100ac5a0 legacy=0x7f57101e7450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.393+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 >> v1:172.21.15.22:6789/0 conn(0x7f571014a700 legacy=0x7f57101ef790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 1 monclient: found mon.b 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:23.378 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571011cfc0 con 0x7f57100b0770 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 10 monclient: _finish_auth 0 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f56feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f571489a640 5 monclient: authenticate success, global_id 4306 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f571489a640 10 monclient: _renew_subs 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f571489a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.394+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f571011e3b0 con 0x7f57100b0770 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.395+0000 7f571489a640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.395+0000 7f571489a640 10 monclient: _renew_subs 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.395+0000 7f571489a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.395+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f571011d9a0 con 0x7f57100b0770 2024-03-29T10:56:23.379 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.395+0000 7f571489a640 1 librados: init done 2024-03-29T10:56:23.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f571489a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:23.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f571489a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:23.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f571489a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:23.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57100ab560 con 0x7f57100b0770 2024-03-29T10:56:23.381 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5704004b00 con 0x7f57100b0770 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f56feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5704004e90 con 0x7f57100b0770 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f56feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f56feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f56feffd640 10 monclient: dump: 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:23.382 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.398+0000 7f570cd99640 4 set_mon_vals no callback set 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.399+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f57040050b0 con 0x7f57100b0770 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.401+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f5704084b10 con 0x7f57100b0770 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.401+0000 7f56feffd640 10 client.4306.objecter ms_dispatch 0x7f5710000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.401+0000 7f56feffd640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.401+0000 7f56feffd640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.401+0000 7f56feffd640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-29T10:56:23.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.401+0000 7f570e59c640 10 monclient: get_auth_request con 0x7f56f006de50 auth_method 0 2024-03-29T10:56:23.384 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.402+0000 7f56feffd640 10 client.4306.objecter ms_handle_connect 0x7f56f006de50 2024-03-29T10:56:23.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.409+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57040503f0 con 0x7f57100b0770 2024-03-29T10:56:23.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.409+0000 7f56feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:23.390 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.409+0000 7f56feffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:23.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.753+0000 7f571489a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:23.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.753+0000 7f571489a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:23.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.753+0000 7f571489a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:23.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.753+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f571011de40 con 0x7f57100b0770 2024-03-29T10:56:23.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.755+0000 7f56feffd640 1 -- 172.21.15.22:0/2870266193 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f5704052320 con 0x7f57100b0770 2024-03-29T10:56:23.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.755+0000 7f56feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:23.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.755+0000 7f56feffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:23.736 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:23.736 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:23.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 10 librados: watch_flush enter 2024-03-29T10:56:23.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 10 librados: watch_flush exit 2024-03-29T10:56:23.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-29T10:56:23.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 10 client.4306.objecter successfully canceled tick 2024-03-29T10:56:23.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 >> v1:172.21.15.22:6816/380221644 conn(0x7f56f006de50 legacy=0x7f56f0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:23.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 10 monclient: shutdown 2024-03-29T10:56:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.758+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 >> v1:172.21.15.26:6789/0 conn(0x7f57100b0770 legacy=0x7f5710147e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.759+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 shutdown_connections 2024-03-29T10:56:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.759+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 >> 172.21.15.22:0/2870266193 conn(0x7f571006b660 msgr2=0x7f571006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.759+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 shutdown_connections 2024-03-29T10:56:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.759+0000 7f571489a640 1 -- 172.21.15.22:0/2870266193 wait complete. 2024-03-29T10:56:23.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:23.759+0000 7f571489a640 1 librados: shutdown 2024-03-29T10:56:23.756 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:30.563 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.741+0000 7f0812d66640 10 monclient: get_monmap_and_config 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.741+0000 7f0812d66640 10 monclient: build_initial_monmap 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.741+0000 7f0812d66640 10 monclient: monmap: 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:30.742170+0000 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:30.742170+0000 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:30.722 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:30.723 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:30.723 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:30.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.742+0000 7f0812d66640 1 Processor -- start 2024-03-29T10:56:30.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.742+0000 7f0812d66640 1 -- start start 2024-03-29T10:56:30.723 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.742+0000 7f0812d66640 10 monclient: init 2024-03-29T10:56:30.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 10 monclient(hunting): picked mon.noname-b con 0x7f080c0ac310 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 10 monclient(hunting): picked mon.noname-a con 0x7f080c0ae460 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 10 monclient(hunting): picked mon.noname-c con 0x7f080c13d270 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c0add30 con 0x7f080c0ae460 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c06df00 con 0x7f080c13d270 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c0ac6f0 con 0x7f080c0ac310 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:30.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.743+0000 7f0812d66640 10 monclient(hunting): authenticate will time out at 892.047696s 2024-03-29T10:56:30.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.746+0000 7f0811563640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f080c0ae460 0x7f080c13cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55887/0 (socket says 172.21.15.22:55887) 2024-03-29T10:56:30.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.746+0000 7f0811563640 1 -- 172.21.15.22:0/4262808858 learned_addr learned my addr 172.21.15.22:0/4262808858 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:30.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.748+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2071274367 0 0) 0x7f080c0add30 con 0x7f080c0ae460 2024-03-29T10:56:30.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): my global_id is 4305 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f8003520 con 0x7f080c0ae460 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3986487183 0 0) 0x7f080c0ac6f0 con 0x7f080c0ac310 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient(hunting): my global_id is 4331 2024-03-29T10:56:30.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f080c0add30 con 0x7f080c0ac310 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4089191550 0 0) 0x7f080c0add30 con 0x7f080c0ac310 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 >> v1:172.21.15.26:6789/0 conn(0x7f080c13d270 legacy=0x7f080c14b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 >> v1:172.21.15.22:6789/0 conn(0x7f080c0ae460 legacy=0x7f080c13cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient: _finish_hunting 0 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 monclient: found mon.noname-b 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c06dc20 con 0x7f080c0ac310 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient: _finish_auth 0 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient: _check_auth_tickets 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:30.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f08000015f0 con 0x7f080c0ac310 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0810d62640 10 monclient: dump: 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:30.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0812d66640 5 monclient: authenticate success, global_id 4331 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.749+0000 7f0812d66640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0800001750 con 0x7f080c0ac310 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0810d62640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0810d62640 1 -- 172.21.15.22:0/4262808858 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0800001b60 con 0x7f080c0ac310 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0810d62640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0810d62640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0810d62640 10 monclient: dump: 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:30.736 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:30.737 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:30.737 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:30.737 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:30.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0812d66640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:30.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.750+0000 7f0812d66640 4 set_mon_vals no callback set 2024-03-29T10:56:30.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.751+0000 7f0812d66640 10 monclient: shutdown 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.751+0000 7f0812d66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.751+0000 7f0812d66640 1 -- 172.21.15.22:0/4262808858 >> v1:172.21.15.22:6790/0 conn(0x7f080c0ac310 legacy=0x7f080c1403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.754+0000 7f0812d66640 1 -- 172.21.15.22:0/4262808858 shutdown_connections 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.754+0000 7f0812d66640 1 -- 172.21.15.22:0/4262808858 >> 172.21.15.22:0/4262808858 conn(0x7f080c0b3a30 msgr2=0x7f080c0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.754+0000 7f0812d66640 1 -- 172.21.15.22:0/4262808858 shutdown_connections 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.754+0000 7f0812d66640 1 -- 172.21.15.22:0/4262808858 wait complete. 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.755+0000 7f0812d66640 10 monclient: build_initial_monmap 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.755+0000 7f0812d66640 10 monclient: monmap: 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:30.738 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.755+0000 7f0812d66640 1 librados: starting msgr at 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.755+0000 7f0812d66640 1 librados: starting objecter 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.756+0000 7f0812d66640 1 Processor -- start 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.756+0000 7f0812d66640 1 -- start start 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.756+0000 7f0812d66640 1 librados: setting wanted keys 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.756+0000 7f0812d66640 1 librados: calling monclient init 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.756+0000 7f0812d66640 10 monclient: init 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:30.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 10 monclient(hunting): picked mon.noname-b con 0x7f080c0ac310 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 10 monclient(hunting): picked mon.noname-a con 0x7f080c0ae460 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 10 monclient(hunting): picked mon.noname-c con 0x7f080c13d270 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c13a540 con 0x7f080c0ae460 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c142320 con 0x7f080c0ac310 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c070cc0 con 0x7f080c13d270 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:30.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.757+0000 7f0812d66640 10 monclient(hunting): authenticate will time out at 892.061437s 2024-03-29T10:56:30.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.761+0000 7f0811d64640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f080c0ac310 0x7f080c071cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:46493/0 (socket says 172.21.15.22:46493) 2024-03-29T10:56:30.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.761+0000 7f0811d64640 1 -- 172.21.15.22:0/3501785732 learned_addr learned my addr 172.21.15.22:0/3501785732 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:30.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 client.?.objecter ms_handle_connect 0x7f080c0ac310 2024-03-29T10:56:30.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:30.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 client.?.objecter ms_handle_connect 0x7f080c13d270 2024-03-29T10:56:30.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:30.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230729960 0 0) 0x7f080c142320 con 0x7f080c0ac310 2024-03-29T10:56:30.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): my global_id is 4309 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07f0004eb0 con 0x7f080c0ac310 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434845607 0 0) 0x7f080c070cc0 con 0x7f080c13d270 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): my global_id is 4334 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f080c142320 con 0x7f080c13d270 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3276515387 0 0) 0x7f07f0004eb0 con 0x7f080c0ac310 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f080c070cc0 con 0x7f080c0ac310 2024-03-29T10:56:30.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0800001b20 con 0x7f080c0ac310 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): dump: 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:30.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.762+0000 7f07fe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3372032262 0 0) 0x7f080c142320 con 0x7f080c13d270 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f07f0004eb0 con 0x7f080c13d270 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0804001b90 con 0x7f080c13d270 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 10 monclient(hunting): dump: 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:30.750 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2124728650 0 0) 0x7f080c070cc0 con 0x7f080c0ac310 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 >> v1:172.21.15.22:6790/0 conn(0x7f080c13d270 legacy=0x7f080c142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 >> v1:172.21.15.22:6789/0 conn(0x7f080c0ae460 legacy=0x7f080c1427d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 monclient: found mon.b 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c070fa0 con 0x7f080c0ac310 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 10 monclient: _finish_auth 0 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:56:30.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f07fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 5 monclient: authenticate success, global_id 4309 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 10 monclient: _renew_subs 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c071280 con 0x7f080c0ac310 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 10 monclient: _renew_subs 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c12f740 con 0x7f080c0ac310 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.763+0000 7f0812d66640 1 librados: init done 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.764+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0800001130 con 0x7f080c0ac310 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.765+0000 7f07fe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.765+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0800003a40 con 0x7f080c0ac310 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.765+0000 7f07fe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.765+0000 7f07fe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:30.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.765+0000 7f07fe7fc640 10 monclient: dump: 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.765+0000 7f07ff7fe640 4 set_mon_vals no callback set 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.765+0000 7f0812d66640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:30.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.766+0000 7f0812d66640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:30.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.766+0000 7f0812d66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:30.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.766+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080c0ad1f0 con 0x7f080c0ac310 2024-03-29T10:56:30.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.766+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f0800003ba0 con 0x7f080c0ac310 2024-03-29T10:56:30.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.767+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0800084b30 con 0x7f080c0ac310 2024-03-29T10:56:30.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.767+0000 7f07fe7fc640 10 client.4309.objecter ms_dispatch 0x7f080c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:30.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.767+0000 7f07fe7fc640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:30.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.767+0000 7f07fe7fc640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:30.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.767+0000 7f0811563640 10 monclient: get_auth_request con 0x7f07f006de40 auth_method 0 2024-03-29T10:56:30.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.767+0000 7f07fe7fc640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T10:56:30.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.768+0000 7f07fe7fc640 10 client.4309.objecter ms_handle_connect 0x7f07f006de40 2024-03-29T10:56:30.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.775+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0800050240 con 0x7f080c0ac310 2024-03-29T10:56:30.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.775+0000 7f07fe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:30.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:30.775+0000 7f07fe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:31.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.098+0000 7f0812d66640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:31.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.098+0000 7f0812d66640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:31.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.098+0000 7f0812d66640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:56:31.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.098+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f080c04f730 con 0x7f080c0ac310 2024-03-29T10:56:31.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.101+0000 7f07fe7fc640 1 -- 172.21.15.22:0/3501785732 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0800052340 con 0x7f080c0ac310 2024-03-29T10:56:31.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.101+0000 7f07fe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:31.082 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.101+0000 7f07fe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:31.082 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:31.082 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:31.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.104+0000 7f0812d66640 10 librados: watch_flush enter 2024-03-29T10:56:31.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 10 librados: watch_flush exit 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 10 client.4309.objecter successfully canceled tick 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 >> v1:172.21.15.22:6816/380221644 conn(0x7f07f006de40 legacy=0x7f07f0070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 10 monclient: shutdown 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 >> v1:172.21.15.26:6789/0 conn(0x7f080c0ac310 legacy=0x7f080c071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 shutdown_connections 2024-03-29T10:56:31.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.105+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 >> 172.21.15.22:0/3501785732 conn(0x7f080c0b3a30 msgr2=0x7f080c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:31.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.107+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 shutdown_connections 2024-03-29T10:56:31.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.107+0000 7f0812d66640 1 -- 172.21.15.22:0/3501785732 wait complete. 2024-03-29T10:56:31.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:31.107+0000 7f0812d66640 1 librados: shutdown 2024-03-29T10:56:31.104 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:37.911 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:38.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.090+0000 7f454da33640 10 monclient: get_monmap_and_config 2024-03-29T10:56:38.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.090+0000 7f454da33640 10 monclient: build_initial_monmap 2024-03-29T10:56:38.071 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.090+0000 7f454da33640 10 monclient: monmap: 2024-03-29T10:56:38.071 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:38.071 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:38.091667+0000 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:38.091667+0000 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:38.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.091+0000 7f454da33640 1 Processor -- start 2024-03-29T10:56:38.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.092+0000 7f454da33640 1 -- start start 2024-03-29T10:56:38.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.092+0000 7f454da33640 10 monclient: init 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.092+0000 7f454da33640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 10 monclient(hunting): picked mon.noname-b con 0x7f454806ade0 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 10 monclient(hunting): picked mon.noname-a con 0x7f454806ed60 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 10 monclient(hunting): picked mon.noname-c con 0x7f4548072340 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454806ff50 con 0x7f454806ed60 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4548070180 con 0x7f4548072340 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f454806f310 con 0x7f454806ade0 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:38.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f454da33640 10 monclient(hunting): authenticate will time out at 899.397263s 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f45467fc640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f454806ed60 0x7f4548071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55646/0 (socket says 172.21.15.22:55646) 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f45467fc640 1 -- 172.21.15.22:0/125768190 learned_addr learned my addr 172.21.15.22:0/125768190 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682844923 0 0) 0x7f454806ff50 con 0x7f454806ed60 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f4545ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f4545ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.093+0000 7f4545ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient(hunting): my global_id is 4308 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4530003520 con 0x7f454806ed60 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797244251 0 0) 0x7f454806f310 con 0x7f454806ade0 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient(hunting): my global_id is 4337 2024-03-29T10:56:38.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f454806ff50 con 0x7f454806ade0 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3877725638 0 0) 0x7f454806ff50 con 0x7f454806ade0 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 >> v1:172.21.15.26:6789/0 conn(0x7f4548072340 legacy=0x7f4548151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 >> v1:172.21.15.22:6789/0 conn(0x7f454806ed60 legacy=0x7f4548071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 monclient: found mon.noname-b 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480aeed0 con 0x7f454806ade0 2024-03-29T10:56:38.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: _finish_auth 0 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4538001670 con 0x7f454806ade0 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: dump: 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:38.078 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f454da33640 5 monclient: authenticate success, global_id 4337 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f454da33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4538001890 con 0x7f454806ade0 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 1 -- 172.21.15.22:0/125768190 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4538001c20 con 0x7f454806ade0 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f4545ffb640 10 monclient: dump: 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.079 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f454da33640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.094+0000 7f454da33640 4 set_mon_vals no callback set 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.095+0000 7f454da33640 10 monclient: shutdown 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.095+0000 7f454da33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.095+0000 7f454da33640 1 -- 172.21.15.22:0/125768190 >> v1:172.21.15.22:6790/0 conn(0x7f454806ade0 legacy=0x7f45481499e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.095+0000 7f454da33640 1 -- 172.21.15.22:0/125768190 shutdown_connections 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.096+0000 7f454da33640 1 -- 172.21.15.22:0/125768190 >> 172.21.15.22:0/125768190 conn(0x7f45480b4930 msgr2=0x7f45480b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:38.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.096+0000 7f454da33640 1 -- 172.21.15.22:0/125768190 shutdown_connections 2024-03-29T10:56:38.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.096+0000 7f454da33640 1 -- 172.21.15.22:0/125768190 wait complete. 2024-03-29T10:56:38.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.097+0000 7f454da33640 10 monclient: build_initial_monmap 2024-03-29T10:56:38.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.097+0000 7f454da33640 10 monclient: monmap: 2024-03-29T10:56:38.085 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:38.085 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.098+0000 7f454da33640 1 librados: starting msgr at 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.098+0000 7f454da33640 1 librados: starting objecter 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.098+0000 7f454da33640 1 Processor -- start 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.098+0000 7f454da33640 1 -- start start 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.098+0000 7f454da33640 1 librados: setting wanted keys 2024-03-29T10:56:38.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.098+0000 7f454da33640 1 librados: calling monclient init 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.098+0000 7f454da33640 10 monclient: init 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.099+0000 7f454da33640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.099+0000 7f454da33640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.099+0000 7f454da33640 10 monclient(hunting): picked mon.noname-b con 0x7f454806ade0 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f454da33640 10 monclient(hunting): picked mon.noname-c con 0x7f454806ed60 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f454da33640 10 monclient(hunting): picked mon.noname-a con 0x7f4548072340 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f454da33640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45481df5e0 con 0x7f4548072340 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f454da33640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45481df300 con 0x7f454806ade0 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f454da33640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45481df030 con 0x7f454806ed60 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f454da33640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f454da33640 10 monclient(hunting): authenticate will time out at 899.404322s 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f45477fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f4548072340 0x7f45480ad5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55842/0 (socket says 172.21.15.22:55842) 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f45477fe640 1 -- 172.21.15.22:0/318604210 learned_addr learned my addr 172.21.15.22:0/318604210 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:38.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 10 client.?.objecter ms_handle_connect 0x7f4548072340 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 10 client.?.objecter ms_handle_connect 0x7f454806ed60 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002766377 0 0) 0x7f45481df030 con 0x7f454806ed60 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.100+0000 7f452ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): my global_id is 4340 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534004eb0 con 0x7f454806ed60 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3471573498 0 0) 0x7f4534004eb0 con 0x7f454806ed60 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45481df030 con 0x7f454806ed60 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f453c001950 con 0x7f454806ed60 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:38.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): dump: 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 855278661 0 0) 0x7f45481df5e0 con 0x7f4548072340 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:56:38.089 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 10 monclient(hunting): my global_id is 4311 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.101+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534004eb0 con 0x7f4548072340 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 375580815 0 0) 0x7f45481df030 con 0x7f454806ed60 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 >> v1:172.21.15.26:6789/0 conn(0x7f454806ade0 legacy=0x7f45481de920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 >> v1:172.21.15.22:6789/0 conn(0x7f4548072340 legacy=0x7f45480ad5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 1 monclient: found mon.c 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45480add00 con 0x7f454806ed60 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: _finish_auth 0 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:56:38.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 5 monclient: authenticate success, global_id 4340 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 10 monclient: _renew_subs 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45480adfb0 con 0x7f454806ed60 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 10 monclient: _renew_subs 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45481e6de0 con 0x7f454806ed60 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f454da33640 1 librados: init done 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f453c0047e0 con 0x7f454806ed60 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f45457fa640 4 set_mon_vals no callback set 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f453c004bb0 con 0x7f454806ed60 2024-03-29T10:56:38.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.102+0000 7f452ffff640 10 monclient: dump: 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:38.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.104+0000 7f454da33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:38.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.104+0000 7f454da33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:38.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.104+0000 7f454da33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:38.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.104+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4528005080 con 0x7f454806ed60 2024-03-29T10:56:38.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.104+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f453c004d10 con 0x7f454806ed60 2024-03-29T10:56:38.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.105+0000 7f4546ffd640 10 monclient: get_auth_request con 0x7f453406dd90 auth_method 0 2024-03-29T10:56:38.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.105+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f453c084b00 con 0x7f454806ed60 2024-03-29T10:56:38.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.105+0000 7f452ffff640 10 client.4340.objecter ms_dispatch 0x7f4548000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:38.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.106+0000 7f452ffff640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:38.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.106+0000 7f452ffff640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:38.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.106+0000 7f452ffff640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-29T10:56:38.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.106+0000 7f452ffff640 10 client.4340.objecter ms_handle_connect 0x7f453406dd90 2024-03-29T10:56:38.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.112+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f453c0503e0 con 0x7f454806ed60 2024-03-29T10:56:38.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.112+0000 7f452ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:38.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.112+0000 7f452ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.535+0000 7f454da33640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.536+0000 7f454da33640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.536+0000 7f454da33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.536+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4528005330 con 0x7f454806ed60 2024-03-29T10:56:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.537+0000 7f452ffff640 1 -- 172.21.15.22:0/318604210 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f453c052310 con 0x7f454806ed60 2024-03-29T10:56:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.537+0000 7f452ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.537+0000 7f452ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:38.518 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:38.518 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:38.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 10 librados: watch_flush enter 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 10 librados: watch_flush exit 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 10 client.4340.objecter successfully canceled tick 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 >> v1:172.21.15.22:6816/380221644 conn(0x7f453406dd90 legacy=0x7f4534070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 10 monclient: shutdown 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 >> v1:172.21.15.22:6790/0 conn(0x7f454806ed60 legacy=0x7f45480acea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 shutdown_connections 2024-03-29T10:56:38.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.540+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 >> 172.21.15.22:0/318604210 conn(0x7f45480b4930 msgr2=0x7f45480afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:38.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.541+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 shutdown_connections 2024-03-29T10:56:38.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.541+0000 7f454da33640 1 -- 172.21.15.22:0/318604210 wait complete. 2024-03-29T10:56:38.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:38.541+0000 7f454da33640 1 librados: shutdown 2024-03-29T10:56:38.536 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:45.343 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:45.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.508+0000 7fa617184640 10 monclient: get_monmap_and_config 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.508+0000 7fa617184640 10 monclient: build_initial_monmap 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.508+0000 7fa617184640 10 monclient: monmap: 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:45.509178+0000 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:45.509178+0000 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:45.489 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.509+0000 7fa617184640 1 Processor -- start 2024-03-29T10:56:45.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.509+0000 7fa617184640 1 -- start start 2024-03-29T10:56:45.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.509+0000 7fa617184640 10 monclient: init 2024-03-29T10:56:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:56:45.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 10 monclient(hunting): picked mon.noname-c con 0x7fa6101457f0 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 10 monclient(hunting): picked mon.noname-b con 0x7fa61013f650 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 10 monclient(hunting): picked mon.noname-a con 0x7fa6100a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6100aaef0 con 0x7fa6100a9d70 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610145bd0 con 0x7fa6101457f0 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610145eb0 con 0x7fa61013f650 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.510+0000 7fa617184640 10 monclient(hunting): authenticate will time out at 906.815060s 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.511+0000 7fa60ffff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa61013f650 0x7fa6100b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41658/0 (socket says 172.21.15.22:41658) 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.511+0000 7fa60ffff640 1 -- 172.21.15.22:0/637198624 learned_addr learned my addr 172.21.15.22:0/637198624 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230822803 0 0) 0x7fa6100aaef0 con 0x7fa6100a9d70 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:45.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): my global_id is 4314 2024-03-29T10:56:45.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f0003520 con 0x7fa6100a9d70 2024-03-29T10:56:45.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86335926 0 0) 0x7fa610145eb0 con 0x7fa61013f650 2024-03-29T10:56:45.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:45.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:45.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:45.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 10 monclient(hunting): my global_id is 4343 2024-03-29T10:56:45.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6100aaef0 con 0x7fa61013f650 2024-03-29T10:56:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.512+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3926267920 0 0) 0x7fa610145bd0 con 0x7fa6101457f0 2024-03-29T10:56:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:56:45.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient(hunting): my global_id is 4312 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa610145eb0 con 0x7fa6101457f0 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2564506883 0 0) 0x7fa5f0003520 con 0x7fa6100a9d70 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 >> v1:172.21.15.22:6790/0 conn(0x7fa61013f650 legacy=0x7fa6100b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 >> v1:172.21.15.26:6789/0 conn(0x7fa6101457f0 legacy=0x7fa61013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 1 monclient: found mon.noname-a 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610146340 con 0x7fa6100a9d70 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:56:45.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa617184640 5 monclient: authenticate success, global_id 4314 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa617184640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa608001690 con 0x7fa6100a9d70 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa60f7fe640 10 monclient: dump: 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:45.500 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.513+0000 7fa617184640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6080018b0 con 0x7fa6100a9d70 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa60f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa60f7fe640 1 -- 172.21.15.22:0/637198624 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa608001c40 con 0x7fa6100a9d70 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa60f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa60f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa60f7fe640 10 monclient: dump: 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.501 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa617184640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.514+0000 7fa617184640 4 set_mon_vals no callback set 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.515+0000 7fa617184640 10 monclient: shutdown 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.515+0000 7fa617184640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.515+0000 7fa617184640 1 -- 172.21.15.22:0/637198624 >> v1:172.21.15.22:6789/0 conn(0x7fa6100a9d70 legacy=0x7fa61014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.516+0000 7fa617184640 1 -- 172.21.15.22:0/637198624 shutdown_connections 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.516+0000 7fa617184640 1 -- 172.21.15.22:0/637198624 >> 172.21.15.22:0/637198624 conn(0x7fa610066a00 msgr2=0x7fa610066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.516+0000 7fa617184640 1 -- 172.21.15.22:0/637198624 shutdown_connections 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.516+0000 7fa617184640 1 -- 172.21.15.22:0/637198624 wait complete. 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.517+0000 7fa617184640 10 monclient: build_initial_monmap 2024-03-29T10:56:45.502 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.517+0000 7fa617184640 10 monclient: monmap: 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.517+0000 7fa617184640 1 librados: starting msgr at 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.517+0000 7fa617184640 1 librados: starting objecter 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.517+0000 7fa617184640 1 Processor -- start 2024-03-29T10:56:45.503 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.517+0000 7fa617184640 1 -- start start 2024-03-29T10:56:45.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.518+0000 7fa617184640 1 librados: setting wanted keys 2024-03-29T10:56:45.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.518+0000 7fa617184640 1 librados: calling monclient init 2024-03-29T10:56:45.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.518+0000 7fa617184640 10 monclient: init 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.519+0000 7fa617184640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.519+0000 7fa617184640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.519+0000 7fa617184640 10 monclient(hunting): picked mon.noname-a con 0x7fa6100a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.519+0000 7fa614f20640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa6100a9d70 0x7fa610070ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:45646/0 (socket says 172.21.15.22:45646) 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.519+0000 7fa614f20640 1 -- 172.21.15.22:0/2744684445 learned_addr learned my addr 172.21.15.22:0/2744684445 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa617184640 10 monclient(hunting): picked mon.noname-c con 0x7fa61013f650 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa60d7fa640 10 client.?.objecter ms_handle_connect 0x7fa6100a9d70 2024-03-29T10:56:45.511 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa60d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa60d7fa640 10 client.?.objecter ms_handle_connect 0x7fa61013f650 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa60d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa617184640 10 monclient(hunting): picked mon.noname-b con 0x7fa6101457f0 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61006fac0 con 0x7fa6100a9d70 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61006fda0 con 0x7fa6101457f0 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610070080 con 0x7fa61013f650 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa617184640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.520+0000 7fa617184640 10 monclient(hunting): authenticate will time out at 906.825062s 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.521+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761209442 0 0) 0x7fa61006fac0 con 0x7fa6100a9d70 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.521+0000 7fa60d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.521+0000 7fa60d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): my global_id is 4317 2024-03-29T10:56:45.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f4004eb0 con 0x7fa6100a9d70 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 client.?.objecter ms_handle_connect 0x7fa6101457f0 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602153671 0 0) 0x7fa610070080 con 0x7fa61013f650 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): my global_id is 4346 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61006fac0 con 0x7fa61013f650 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3468525275 0 0) 0x7fa61006fda0 con 0x7fa6101457f0 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:45.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 10 monclient(hunting): my global_id is 4315 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.522+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa610070080 con 0x7fa6101457f0 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4202376152 0 0) 0x7fa61006fac0 con 0x7fa61013f650 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa61006fda0 con 0x7fa61013f650 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1491143523 0 0) 0x7fa610070080 con 0x7fa6101457f0 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa61006fac0 con 0x7fa6101457f0 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1629460522 0 0) 0x7fa5f4004eb0 con 0x7fa6100a9d70 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa610070080 con 0x7fa6100a9d70 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa5f8001c30 con 0x7fa61013f650 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:45.514 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa608001430 con 0x7fa6101457f0 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:45.515 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa600001890 con 0x7fa6100a9d70 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient(hunting): dump: 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:45.516 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2053761347 0 0) 0x7fa610070080 con 0x7fa6100a9d70 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 >> v1:172.21.15.22:6790/0 conn(0x7fa61013f650 legacy=0x7fa6101394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 >> v1:172.21.15.26:6789/0 conn(0x7fa6101457f0 legacy=0x7fa61006df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 monclient: found mon.a 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:45.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610070360 con 0x7fa6100a9d70 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa60d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa617184640 5 monclient: authenticate success, global_id 4317 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa617184640 10 monclient: _renew_subs 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa617184640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.523+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa610071380 con 0x7fa6100a9d70 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.524+0000 7fa617184640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.524+0000 7fa617184640 10 monclient: _renew_subs 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.524+0000 7fa617184640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.524+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa610139dd0 con 0x7fa6100a9d70 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.524+0000 7fa617184640 1 librados: init done 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa617184640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:45.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa617184640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa617184640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6100711e0 con 0x7fa6100a9d70 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6000035f0 con 0x7fa6100a9d70 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa60d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa600001d80 con 0x7fa6100a9d70 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa60d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa60d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa60d7fa640 10 monclient: dump: 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:45.519 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.525+0000 7fa60effd640 4 set_mon_vals no callback set 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.527+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fa600003920 con 0x7fa6100a9d70 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.528+0000 7fa60ffff640 10 monclient: get_auth_request con 0x7fa5f406e1b0 auth_method 0 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.528+0000 7fa60d7fa640 10 client.4317.objecter ms_handle_connect 0x7fa5f406e1b0 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.528+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fa600084740 con 0x7fa6100a9d70 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.528+0000 7fa60d7fa640 10 client.4317.objecter ms_dispatch 0x7fa610000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.528+0000 7fa60d7fa640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.528+0000 7fa60d7fa640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:45.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.529+0000 7fa60d7fa640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-29T10:56:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.535+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa600050020 con 0x7fa6100a9d70 2024-03-29T10:56:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.535+0000 7fa60d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:45.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.535+0000 7fa60d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:45.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.938+0000 7fa617184640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:45.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.938+0000 7fa617184640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:45.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.938+0000 7fa617184640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:56:45.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.938+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa610071510 con 0x7fa6100a9d70 2024-03-29T10:56:45.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.939+0000 7fa60d7fa640 1 -- 172.21.15.22:0/2744684445 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fa600051f50 con 0x7fa6100a9d70 2024-03-29T10:56:45.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.939+0000 7fa60d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:45.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.939+0000 7fa60d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:45.920 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:45.920 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:45.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.942+0000 7fa617184640 10 librados: watch_flush enter 2024-03-29T10:56:45.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.942+0000 7fa617184640 10 librados: watch_flush exit 2024-03-29T10:56:45.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.942+0000 7fa617184640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-29T10:56:45.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.942+0000 7fa617184640 10 client.4317.objecter successfully canceled tick 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 >> v1:172.21.15.22:6816/380221644 conn(0x7fa5f406e1b0 legacy=0x7fa5f4070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 10 monclient: shutdown 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 >> v1:172.21.15.22:6789/0 conn(0x7fa6100a9d70 legacy=0x7fa610070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 shutdown_connections 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 >> 172.21.15.22:0/2744684445 conn(0x7fa610066a00 msgr2=0x7fa6101d2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 shutdown_connections 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 1 -- 172.21.15.22:0/2744684445 wait complete. 2024-03-29T10:56:45.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:45.943+0000 7fa617184640 1 librados: shutdown 2024-03-29T10:56:45.938 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:56:52.745 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.911+0000 7fed6f092640 10 monclient: get_monmap_and_config 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.911+0000 7fed6f092640 10 monclient: build_initial_monmap 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.911+0000 7fed6f092640 10 monclient: monmap: 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:56:52.912639+0000 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:56:52.912639+0000 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:52.892 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:52.893 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:52.893 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:56:52.893 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:56:52.893 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:52.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.912+0000 7fed6f092640 1 Processor -- start 2024-03-29T10:56:52.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.912+0000 7fed6f092640 1 -- start start 2024-03-29T10:56:52.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.912+0000 7fed6f092640 10 monclient: init 2024-03-29T10:56:52.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.913+0000 7fed6f092640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:52.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.913+0000 7fed6f092640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:56:52.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.913+0000 7fed6f092640 10 monclient(hunting): picked mon.noname-b con 0x7fed60036a60 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:52.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.913+0000 7fed6f092640 10 monclient(hunting): picked mon.noname-c con 0x7fed600c5de0 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:52.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.913+0000 7fed6f092640 10 monclient(hunting): picked mon.noname-a con 0x7fed600c3070 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:52.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.914+0000 7fed6f092640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6002cc60 con 0x7fed600c3070 2024-03-29T10:56:52.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.914+0000 7fed6f092640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6002e040 con 0x7fed600c5de0 2024-03-29T10:56:52.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.914+0000 7fed6f092640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6002e320 con 0x7fed60036a60 2024-03-29T10:56:52.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.914+0000 7fed6f092640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:52.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.914+0000 7fed6f092640 10 monclient(hunting): authenticate will time out at 914.218114s 2024-03-29T10:56:52.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.914+0000 7fed6e090640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fed60036a60 0x7fed600c56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49742/0 (socket says 172.21.15.22:49742) 2024-03-29T10:56:52.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.914+0000 7fed6e090640 1 -- 172.21.15.22:0/2121382773 learned_addr learned my addr 172.21.15.22:0/2121382773 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30528334 0 0) 0x7fed6002e320 con 0x7fed60036a60 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): _negotiate 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): my global_id is 4349 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed54003520 con 0x7fed60036a60 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 391643420 0 0) 0x7fed6002e040 con 0x7fed600c5de0 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): _negotiate 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:52.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:52.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 10 monclient(hunting): my global_id is 4318 2024-03-29T10:56:52.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.917+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6002e320 con 0x7fed600c5de0 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597496974 0 0) 0x7fed6002cc60 con 0x7fed600c3070 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient(hunting): _negotiate 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient(hunting): my global_id is 4320 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed6002e040 con 0x7fed600c3070 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1808354587 0 0) 0x7fed54003520 con 0x7fed60036a60 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 >> v1:172.21.15.26:6789/0 conn(0x7fed600c5de0 legacy=0x7fed600c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 >> v1:172.21.15.22:6789/0 conn(0x7fed600c3070 legacy=0x7fed600d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: _finish_hunting 0 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 monclient: found mon.noname-b 2024-03-29T10:56:52.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed600095c0 con 0x7fed60036a60 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: _finish_auth 0 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: _check_auth_tickets 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fed58001610 con 0x7fed60036a60 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: dump: 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:52.901 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed58001770 con 0x7fed60036a60 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 1 -- 172.21.15.22:0/2121382773 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fed58001bc0 con 0x7fed60036a60 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.919+0000 7fed6d08e640 10 monclient: dump: 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.902 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.903 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:52.903 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:52.903 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:52.903 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:52.903 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:52.903 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.921+0000 7fed6f092640 5 monclient: authenticate success, global_id 4349 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.921+0000 7fed6f092640 10 monclient: get_monmap_and_config success 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.921+0000 7fed6f092640 4 set_mon_vals no callback set 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.922+0000 7fed6f092640 10 monclient: shutdown 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.922+0000 7fed6f092640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.922+0000 7fed6f092640 1 -- 172.21.15.22:0/2121382773 >> v1:172.21.15.22:6790/0 conn(0x7fed60036a60 legacy=0x7fed600c56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.922+0000 7fed6f092640 1 -- 172.21.15.22:0/2121382773 shutdown_connections 2024-03-29T10:56:52.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.922+0000 7fed6f092640 1 -- 172.21.15.22:0/2121382773 >> 172.21.15.22:0/2121382773 conn(0x7fed60005c00 msgr2=0x7fed60006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.922+0000 7fed6f092640 1 -- 172.21.15.22:0/2121382773 shutdown_connections 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.923+0000 7fed6f092640 1 -- 172.21.15.22:0/2121382773 wait complete. 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.923+0000 7fed6f092640 10 monclient: build_initial_monmap 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.923+0000 7fed6f092640 10 monclient: monmap: 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:56:52.905 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:52.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.924+0000 7fed6f092640 1 librados: starting msgr at 2024-03-29T10:56:52.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.924+0000 7fed6f092640 1 librados: starting objecter 2024-03-29T10:56:52.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.924+0000 7fed6f092640 1 Processor -- start 2024-03-29T10:56:52.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.924+0000 7fed6f092640 1 -- start start 2024-03-29T10:56:52.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.924+0000 7fed6f092640 1 librados: setting wanted keys 2024-03-29T10:56:52.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.924+0000 7fed6f092640 1 librados: calling monclient init 2024-03-29T10:56:52.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.924+0000 7fed6f092640 10 monclient: init 2024-03-29T10:56:52.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 10 monclient: _reopen_session rank -1 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 10 monclient(hunting): picked mon.noname-c con 0x7fed60036a60 addr v1:172.21.15.22:6790/0 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 10 monclient(hunting): picked mon.noname-a con 0x7fed600c3070 addr v1:172.21.15.22:6789/0 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 10 monclient(hunting): picked mon.noname-b con 0x7fed60031370 addr v1:172.21.15.26:6789/0 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed600bec40 con 0x7fed600c3070 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed600bff00 con 0x7fed60031370 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6e090640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fed60036a60 0x7fed6002eaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50010/0 (socket says 172.21.15.22:50010) 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6e090640 1 -- 172.21.15.22:0/2534991481 learned_addr learned my addr 172.21.15.22:0/2534991481 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed60030f50 con 0x7fed60036a60 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 10 monclient(hunting): _renew_subs 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed6f092640 10 monclient(hunting): authenticate will time out at 914.229702s 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 10 client.?.objecter ms_handle_connect 0x7fed60036a60 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:52.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1002693617 0 0) 0x7fed60030f50 con 0x7fed60036a60 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 10 monclient(hunting): my global_id is 4352 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.925+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed4c004eb0 con 0x7fed60036a60 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3364582031 0 0) 0x7fed4c004eb0 con 0x7fed60036a60 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed60030f50 con 0x7fed60036a60 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fed58001b00 con 0x7fed60036a60 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 10 monclient(hunting): dump: 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:52.909 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2918695865 0 0) 0x7fed60030f50 con 0x7fed60036a60 2024-03-29T10:56:52.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.926+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 >> v1:172.21.15.26:6789/0 conn(0x7fed60031370 legacy=0x7fed60167880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 >> v1:172.21.15.22:6789/0 conn(0x7fed600c3070 legacy=0x7fed600be530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 10 monclient: _finish_hunting 0 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 1 monclient: found mon.c 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60031800 con 0x7fed60036a60 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 10 monclient: _finish_auth 0 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 10 monclient: _check_auth_tickets 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 10 client.?.objecter ms_handle_connect 0x7fed600c3070 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:56:52.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed6f092640 5 monclient: authenticate success, global_id 4352 2024-03-29T10:56:52.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed6f092640 10 monclient: _renew_subs 2024-03-29T10:56:52.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed6f092640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:52.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6002f200 con 0x7fed60036a60 2024-03-29T10:56:52.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed58004a90 con 0x7fed60036a60 2024-03-29T10:56:52.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.931+0000 7fed5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:56:52.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fed58004ee0 con 0x7fed60036a60 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed5effd640 10 monclient: dump: 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:56:52.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed6c88d640 4 set_mon_vals no callback set 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed6f092640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed6f092640 10 monclient: _renew_subs 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed6f092640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed600bca70 con 0x7fed60036a60 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.932+0000 7fed6f092640 1 librados: init done 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.934+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fed5801aeb0 con 0x7fed60036a60 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.935+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fed58084d60 con 0x7fed60036a60 2024-03-29T10:56:52.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.935+0000 7fed5effd640 10 client.4352.objecter ms_dispatch 0x7fed68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:56:52.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.935+0000 7fed5effd640 3 client.4352.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:56:52.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.935+0000 7fed5effd640 3 client.4352.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:56:52.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.935+0000 7fed5effd640 20 client.4352.objecter dump_active .. 0 homeless 2024-03-29T10:56:52.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.935+0000 7fed6d88f640 10 monclient: get_auth_request con 0x7fed4c06db80 auth_method 0 2024-03-29T10:56:52.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.935+0000 7fed6f092640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:52.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.936+0000 7fed6f092640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:52.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.936+0000 7fed6f092640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:52.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.936+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed60005120 con 0x7fed60036a60 2024-03-29T10:56:52.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.940+0000 7fed5effd640 10 client.4352.objecter ms_handle_connect 0x7fed4c06db80 2024-03-29T10:56:52.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.945+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed58050640 con 0x7fed60036a60 2024-03-29T10:56:52.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.945+0000 7fed5effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:56:52.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:52.945+0000 7fed5effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:56:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.275+0000 7fed6f092640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.275+0000 7fed6f092640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.275+0000 7fed6f092640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:56:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.275+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed600c0150 con 0x7fed60036a60 2024-03-29T10:56:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.276+0000 7fed5effd640 1 -- 172.21.15.22:0/2534991481 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fed58052570 con 0x7fed60036a60 2024-03-29T10:56:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.276+0000 7fed5effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:56:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.276+0000 7fed5effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:56:53.257 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:56:53.257 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:56:53.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.280+0000 7fed6f092640 10 librados: watch_flush enter 2024-03-29T10:56:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.280+0000 7fed6f092640 10 librados: watch_flush exit 2024-03-29T10:56:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.280+0000 7fed6f092640 20 client.4352.objecter shutdown clearing up homeless session... 2024-03-29T10:56:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.280+0000 7fed6f092640 10 client.4352.objecter successfully canceled tick 2024-03-29T10:56:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.281+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 >> v1:172.21.15.22:6816/380221644 conn(0x7fed4c06db80 legacy=0x7fed4c070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.281+0000 7fed6f092640 10 monclient: shutdown 2024-03-29T10:56:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.281+0000 7fed6f092640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:56:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.281+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 >> v1:172.21.15.22:6790/0 conn(0x7fed60036a60 legacy=0x7fed6002eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:56:53.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.283+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 shutdown_connections 2024-03-29T10:56:53.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.283+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 >> 172.21.15.22:0/2534991481 conn(0x7fed60005c00 msgr2=0x7fed600c2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:56:53.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.283+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 shutdown_connections 2024-03-29T10:56:53.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.283+0000 7fed6f092640 1 -- 172.21.15.22:0/2534991481 wait complete. 2024-03-29T10:56:53.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:56:53.284+0000 7fed6f092640 1 librados: shutdown 2024-03-29T10:56:53.280 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:00.088 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:00.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.238+0000 7fe8d4055640 10 monclient: get_monmap_and_config 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.238+0000 7fe8d4055640 10 monclient: build_initial_monmap 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.238+0000 7fe8d4055640 10 monclient: monmap: 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:00.239304+0000 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:00.239304+0000 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:00.219 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:00.220 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:00.220 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:00.220 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:00.220 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:00.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.239+0000 7fe8d4055640 1 Processor -- start 2024-03-29T10:57:00.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.239+0000 7fe8d4055640 1 -- start start 2024-03-29T10:57:00.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.239+0000 7fe8d4055640 10 monclient: init 2024-03-29T10:57:00.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.240+0000 7fe8d4055640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:00.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.240+0000 7fe8d4055640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.240+0000 7fe8d4055640 10 monclient(hunting): picked mon.noname-b con 0x7fe8cc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.240+0000 7fe8d1df1640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe8cc1457f0 0x7fe8cc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:33812/0 (socket says 172.21.15.22:33812) 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.240+0000 7fe8d1df1640 1 -- 172.21.15.22:0/3864449262 learned_addr learned my addr 172.21.15.22:0/3864449262 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.240+0000 7fe8d4055640 10 monclient(hunting): picked mon.noname-a con 0x7fe8cc13f650 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.241+0000 7fe8d4055640 10 monclient(hunting): picked mon.noname-c con 0x7fe8cc0a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.241+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc0aaef0 con 0x7fe8cc13f650 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.241+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc145bd0 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.241+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc145eb0 con 0x7fe8cc1457f0 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.241+0000 7fe8d4055640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:00.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.241+0000 7fe8d4055640 10 monclient(hunting): authenticate will time out at 921.545554s 2024-03-29T10:57:00.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689024428 0 0) 0x7fe8cc145eb0 con 0x7fe8cc1457f0 2024-03-29T10:57:00.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): _negotiate 2024-03-29T10:57:00.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:00.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:00.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): my global_id is 4355 2024-03-29T10:57:00.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b8003520 con 0x7fe8cc1457f0 2024-03-29T10:57:00.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059638723 0 0) 0x7fe8cc145bd0 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): _negotiate 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 10 monclient(hunting): my global_id is 4321 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8cc145eb0 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3227460886 0 0) 0x7fe8b8003520 con 0x7fe8cc1457f0 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 >> v1:172.21.15.26:6789/0 conn(0x7fe8cc0a9d70 legacy=0x7fe8cc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.242+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 >> v1:172.21.15.22:6789/0 conn(0x7fe8cc13f650 legacy=0x7fe8cc0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 10 monclient: _finish_hunting 0 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 1 monclient: found mon.noname-b 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc146340 con 0x7fe8cc1457f0 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 10 monclient: _finish_auth 0 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 10 monclient: _check_auth_tickets 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:00.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe8c8001690 con 0x7fe8cc1457f0 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d4055640 5 monclient: authenticate success, global_id 4355 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d4055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d0def640 10 monclient: dump: 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:00.228 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.244+0000 7fe8d4055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8c80017f0 con 0x7fe8cc1457f0 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d0def640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d0def640 1 -- 172.21.15.22:0/3864449262 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe8c8001c00 con 0x7fe8cc1457f0 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d0def640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d0def640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d0def640 10 monclient: dump: 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:00.229 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d4055640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d4055640 4 set_mon_vals no callback set 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d4055640 10 monclient: shutdown 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d4055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:00.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.245+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 >> v1:172.21.15.22:6790/0 conn(0x7fe8cc1457f0 legacy=0x7fe8cc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.246+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 shutdown_connections 2024-03-29T10:57:00.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.246+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 >> 172.21.15.22:0/3864449262 conn(0x7fe8cc066a00 msgr2=0x7fe8cc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:00.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.246+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 shutdown_connections 2024-03-29T10:57:00.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.246+0000 7fe8d4055640 1 -- 172.21.15.22:0/3864449262 wait complete. 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.247+0000 7fe8d4055640 10 monclient: build_initial_monmap 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.247+0000 7fe8d4055640 10 monclient: monmap: 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:00.241 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.248+0000 7fe8d4055640 1 librados: starting msgr at 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.248+0000 7fe8d4055640 1 librados: starting objecter 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.248+0000 7fe8d4055640 1 Processor -- start 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.248+0000 7fe8d4055640 1 -- start start 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.248+0000 7fe8d4055640 1 librados: setting wanted keys 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.248+0000 7fe8d4055640 1 librados: calling monclient init 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.248+0000 7fe8d4055640 10 monclient: init 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:00.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 10 monclient(hunting): picked mon.noname-c con 0x7fe8cc0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 10 monclient(hunting): picked mon.noname-b con 0x7fe8cc13f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 10 monclient(hunting): picked mon.noname-a con 0x7fe8cc1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc06fb60 con 0x7fe8cc1457f0 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc0711b0 con 0x7fe8cc13f650 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8cc071490 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.249+0000 7fe8d4055640 10 monclient(hunting): authenticate will time out at 921.553760s 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.250+0000 7fe8d15f0640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe8cc13f650 0x7fe8cc139830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:42736/0 (socket says 172.21.15.22:42736) 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.250+0000 7fe8d15f0640 1 -- 172.21.15.22:0/924773292 learned_addr learned my addr 172.21.15.22:0/924773292 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.250+0000 7fe8c27fc640 10 client.?.objecter ms_handle_connect 0x7fe8cc13f650 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.250+0000 7fe8c27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:00.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.250+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614862506 0 0) 0x7fe8cc0711b0 con 0x7fe8cc13f650 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): my global_id is 4324 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b0004eb0 con 0x7fe8cc13f650 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 client.?.objecter ms_handle_connect 0x7fe8cc0a9d70 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982560916 0 0) 0x7fe8cc071490 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 monclient(hunting): my global_id is 4358 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8cc0711b0 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 client.?.objecter ms_handle_connect 0x7fe8cc1457f0 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.251+0000 7fe8c27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1556607059 0 0) 0x7fe8b0004eb0 con 0x7fe8cc13f650 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8cc071490 con 0x7fe8cc13f650 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014677715 0 0) 0x7fe8cc06fb60 con 0x7fe8cc1457f0 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): my global_id is 4326 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe8b0004eb0 con 0x7fe8cc1457f0 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe8bc001960 con 0x7fe8cc13f650 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): dump: 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:00.245 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.252+0000 7fe8c27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778561654 0 0) 0x7fe8b0004eb0 con 0x7fe8cc1457f0 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8cc06fb60 con 0x7fe8cc1457f0 2024-03-29T10:57:00.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1498826384 0 0) 0x7fe8cc0711b0 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8b0004eb0 con 0x7fe8cc0a9d70 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2556120217 0 0) 0x7fe8cc071490 con 0x7fe8cc13f650 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 >> v1:172.21.15.22:6790/0 conn(0x7fe8cc0a9d70 legacy=0x7fe8cc1db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 >> v1:172.21.15.22:6789/0 conn(0x7fe8cc1457f0 legacy=0x7fe8cc1e6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 monclient: found mon.b 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8cc06f760 con 0x7fe8cc13f650 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:00.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 5 monclient: authenticate success, global_id 4324 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 10 monclient: _renew_subs 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8cc06fff0 con 0x7fe8cc13f650 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 10 monclient: _renew_subs 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8cc06f010 con 0x7fe8cc13f650 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.253+0000 7fe8d4055640 1 librados: init done 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.254+0000 7fe8d4055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.254+0000 7fe8d4055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.254+0000 7fe8d4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.254+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8cc075f10 con 0x7fe8cc13f650 2024-03-29T10:57:00.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.255+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8bc001ac0 con 0x7fe8cc13f650 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.255+0000 7fe8c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.255+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe8bc0038a0 con 0x7fe8cc13f650 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.255+0000 7fe8c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.255+0000 7fe8c27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.255+0000 7fe8c27fc640 10 monclient: dump: 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:00.249 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.257+0000 7fe8c3fff640 4 set_mon_vals no callback set 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.257+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fe8bc003ac0 con 0x7fe8cc13f650 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.258+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fe8bc084950 con 0x7fe8cc13f650 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.258+0000 7fe8d1df1640 10 monclient: get_auth_request con 0x7fe8b006df30 auth_method 0 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.258+0000 7fe8c27fc640 10 client.4324.objecter ms_dispatch 0x7fe8cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.258+0000 7fe8c27fc640 3 client.4324.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.258+0000 7fe8c27fc640 3 client.4324.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.258+0000 7fe8c27fc640 20 client.4324.objecter dump_active .. 0 homeless 2024-03-29T10:57:00.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.258+0000 7fe8c27fc640 10 client.4324.objecter ms_handle_connect 0x7fe8b006df30 2024-03-29T10:57:00.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.263+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8bc050230 con 0x7fe8cc13f650 2024-03-29T10:57:00.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.263+0000 7fe8c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:00.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.263+0000 7fe8c27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:00.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.587+0000 7fe8d4055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:00.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.587+0000 7fe8d4055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:00.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.587+0000 7fe8d4055640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:00.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.587+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8cc0b3680 con 0x7fe8cc13f650 2024-03-29T10:57:00.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.590+0000 7fe8c27fc640 1 -- 172.21.15.22:0/924773292 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fe8bc052160 con 0x7fe8cc13f650 2024-03-29T10:57:00.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.590+0000 7fe8c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:00.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.590+0000 7fe8c27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:00.571 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:00.571 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:00.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.593+0000 7fe8d4055640 10 librados: watch_flush enter 2024-03-29T10:57:00.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.594+0000 7fe8d4055640 10 librados: watch_flush exit 2024-03-29T10:57:00.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.594+0000 7fe8d4055640 20 client.4324.objecter shutdown clearing up homeless session... 2024-03-29T10:57:00.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.594+0000 7fe8d4055640 10 client.4324.objecter successfully canceled tick 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 >> v1:172.21.15.22:6816/380221644 conn(0x7fe8b006df30 legacy=0x7fe8b00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 10 monclient: shutdown 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 >> v1:172.21.15.26:6789/0 conn(0x7fe8cc13f650 legacy=0x7fe8cc139830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 shutdown_connections 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 >> 172.21.15.22:0/924773292 conn(0x7fe8cc066a00 msgr2=0x7fe8cc0b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 shutdown_connections 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.595+0000 7fe8d4055640 1 -- 172.21.15.22:0/924773292 wait complete. 2024-03-29T10:57:00.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:00.596+0000 7fe8d4055640 1 librados: shutdown 2024-03-29T10:57:00.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:07.400 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.572+0000 7f4ef8f8f640 10 monclient: get_monmap_and_config 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.572+0000 7f4ef8f8f640 10 monclient: build_initial_monmap 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.572+0000 7f4ef8f8f640 10 monclient: monmap: 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:07.573223+0000 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:07.573223+0000 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:07.553 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:07.554 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:07.554 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:07.554 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:07.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.573+0000 7f4ef8f8f640 1 Processor -- start 2024-03-29T10:57:07.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.573+0000 7f4ef8f8f640 1 -- start start 2024-03-29T10:57:07.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.573+0000 7f4ef8f8f640 10 monclient: init 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef406d9d0 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef413e820 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef40af1d0 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40aed10 con 0x7f4ef40af1d0 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40afd50 con 0x7f4ef406d9d0 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef40ab490 con 0x7f4ef413e820 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef8f8f640 10 monclient(hunting): authenticate will time out at 928.878499s 2024-03-29T10:57:07.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef1d9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4ef413e820 0x7f4ef413ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47068/0 (socket says 172.21.15.22:47068) 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef1d9b640 1 -- 172.21.15.22:0/306635564 learned_addr learned my addr 172.21.15.22:0/306635564 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662908685 0 0) 0x7f4ef40ab490 con 0x7f4ef413e820 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef159a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef159a640 10 monclient(hunting): my global_id is 4361 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.574+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4edc00bd30 con 0x7f4ef413e820 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.575+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1634236501 0 0) 0x7f4edc00bd30 con 0x7f4ef413e820 2024-03-29T10:57:07.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.575+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 >> v1:172.21.15.26:6789/0 conn(0x7f4ef406d9d0 legacy=0x7f4ef40b05f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 >> v1:172.21.15.22:6789/0 conn(0x7f4ef40af1d0 legacy=0x7f4ef40af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 1 monclient: found mon.noname-b 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef40ab770 con 0x7f4ef413e820 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 10 monclient: _finish_auth 0 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:07.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4ee8001690 con 0x7f4ef413e820 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.582+0000 7f4ef159a640 10 monclient: dump: 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:07.564 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef8f8f640 5 monclient: authenticate success, global_id 4361 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef8f8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ee8001920 con 0x7f4ef413e820 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef159a640 1 -- 172.21.15.22:0/306635564 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4ee8001cb0 con 0x7f4ef413e820 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef159a640 10 monclient: dump: 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.566 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef8f8f640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.583+0000 7f4ef8f8f640 4 set_mon_vals no callback set 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.584+0000 7f4ef8f8f640 10 monclient: shutdown 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.584+0000 7f4ef8f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.584+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/306635564 >> v1:172.21.15.22:6790/0 conn(0x7f4ef413e820 legacy=0x7f4ef413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.584+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/306635564 shutdown_connections 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.584+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/306635564 >> 172.21.15.22:0/306635564 conn(0x7f4ef406b660 msgr2=0x7f4ef406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:07.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.584+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/306635564 shutdown_connections 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.586+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/306635564 wait complete. 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.587+0000 7f4ef8f8f640 10 monclient: build_initial_monmap 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.587+0000 7f4ef8f8f640 10 monclient: monmap: 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:07.568 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:07.569 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:07.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.588+0000 7f4ef8f8f640 1 librados: starting msgr at 2024-03-29T10:57:07.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.588+0000 7f4ef8f8f640 1 librados: starting objecter 2024-03-29T10:57:07.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.588+0000 7f4ef8f8f640 1 Processor -- start 2024-03-29T10:57:07.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.588+0000 7f4ef8f8f640 1 -- start start 2024-03-29T10:57:07.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.588+0000 7f4ef8f8f640 1 librados: setting wanted keys 2024-03-29T10:57:07.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.588+0000 7f4ef8f8f640 1 librados: calling monclient init 2024-03-29T10:57:07.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.588+0000 7f4ef8f8f640 10 monclient: init 2024-03-29T10:57:07.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.589+0000 7f4ef8f8f640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:07.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.589+0000 7f4ef8f8f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:57:07.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.589+0000 7f4ef8f8f640 10 monclient(hunting): picked mon.noname-c con 0x7f4ef406d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:07.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef259c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4ef406d9d0 0x7f4ef40715c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47256/0 (socket says 172.21.15.22:47256) 2024-03-29T10:57:07.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef8f8f640 10 monclient(hunting): picked mon.noname-b con 0x7f4ef40af1d0 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:07.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef8f8f640 10 monclient(hunting): picked mon.noname-a con 0x7f4ef413e820 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef8f8f640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef4071cd0 con 0x7f4ef413e820 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef8f8f640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef41e41a0 con 0x7f4ef40af1d0 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef8f8f640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ef4070b50 con 0x7f4ef406d9d0 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef8f8f640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef8f8f640 10 monclient(hunting): authenticate will time out at 928.895650s 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ef259c640 1 -- 172.21.15.22:0/3910604634 learned_addr learned my addr 172.21.15.22:0/3910604634 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ee2ffd640 10 client.?.objecter ms_handle_connect 0x7f4ef406d9d0 2024-03-29T10:57:07.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.591+0000 7f4ee2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 client.?.objecter ms_handle_connect 0x7f4ef40af1d0 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121999946 0 0) 0x7f4ef4070b50 con 0x7f4ef406d9d0 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): my global_id is 4364 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ed4004eb0 con 0x7f4ef406d9d0 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2541107714 0 0) 0x7f4ef41e41a0 con 0x7f4ef40af1d0 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): my global_id is 4327 2024-03-29T10:57:07.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef4070b50 con 0x7f4ef40af1d0 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3867525236 0 0) 0x7f4ed4004eb0 con 0x7f4ef406d9d0 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ef41e41a0 con 0x7f4ef406d9d0 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4ee40017d0 con 0x7f4ef406d9d0 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.574 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.592+0000 7f4ee2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.593+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 508520821 0 0) 0x7f4ef4070b50 con 0x7f4ef40af1d0 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.593+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ed4004eb0 con 0x7f4ef40af1d0 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.593+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4ee8001550 con 0x7f4ef40af1d0 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.593+0000 7f4ee2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.593+0000 7f4ee2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.593+0000 7f4ee2ffd640 10 monclient(hunting): dump: 2024-03-29T10:57:07.575 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:07.576 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.594+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2197104818 0 0) 0x7f4ed4004eb0 con 0x7f4ef40af1d0 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.594+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 >> v1:172.21.15.22:6790/0 conn(0x7f4ef406d9d0 legacy=0x7f4ef40715c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.594+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 >> v1:172.21.15.22:6789/0 conn(0x7f4ef413e820 legacy=0x7f4ef41e3340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 1 monclient: found mon.b 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef41e4d90 con 0x7f4ef40af1d0 2024-03-29T10:57:07.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ee80016b0 con 0x7f4ef40af1d0 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ee2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.595+0000 7f4ef0d99640 4 set_mon_vals no callback set 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 5 monclient: authenticate success, global_id 4327 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 10 monclient: _renew_subs 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef406e760 con 0x7f4ef40af1d0 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 10 monclient: _renew_subs 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef4142d50 con 0x7f4ef40af1d0 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ef8f8f640 1 librados: init done 2024-03-29T10:57:07.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.596+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4ee8004d90 con 0x7f4ef40af1d0 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.597+0000 7f4ee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.597+0000 7f4ee2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.597+0000 7f4ee2ffd640 10 monclient: dump: 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:07.580 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.599+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f4ee801ad60 con 0x7f4ef40af1d0 2024-03-29T10:57:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.599+0000 7f4ef8f8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.599+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f4ee8084ab0 con 0x7f4ef40af1d0 2024-03-29T10:57:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.599+0000 7f4ee2ffd640 10 client.4327.objecter ms_dispatch 0x7f4ef4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:07.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.599+0000 7f4ee2ffd640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.599+0000 7f4ee2ffd640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.599+0000 7f4ee2ffd640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-29T10:57:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.600+0000 7f4ef8f8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.600+0000 7f4ef8f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.600+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef41e4000 con 0x7f4ef40af1d0 2024-03-29T10:57:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.600+0000 7f4ef259c640 10 monclient: get_auth_request con 0x7f4ed406dd70 auth_method 0 2024-03-29T10:57:07.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.600+0000 7f4ee2ffd640 10 client.4327.objecter ms_handle_connect 0x7f4ed406dd70 2024-03-29T10:57:07.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.610+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ef41e4000 con 0x7f4ef40af1d0 2024-03-29T10:57:07.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.611+0000 7f4ee2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:07.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.611+0000 7f4ee2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:07.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.950+0000 7f4ef8f8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:07.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.950+0000 7f4ef8f8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:07.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.950+0000 7f4ef8f8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:57:07.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.950+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ef41e4f20 con 0x7f4ef40af1d0 2024-03-29T10:57:07.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.953+0000 7f4ee2ffd640 1 -- 172.21.15.22:0/3910604634 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f4ef41e4f20 con 0x7f4ef40af1d0 2024-03-29T10:57:07.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.953+0000 7f4ee2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:07.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.953+0000 7f4ee2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:07.935 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:07.935 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:07.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 10 librados: watch_flush enter 2024-03-29T10:57:07.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 10 librados: watch_flush exit 2024-03-29T10:57:07.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-29T10:57:07.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 10 client.4327.objecter successfully canceled tick 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 >> v1:172.21.15.22:6816/380221644 conn(0x7f4ed406dd70 legacy=0x7f4ed4070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 10 monclient: shutdown 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 >> v1:172.21.15.26:6789/0 conn(0x7f4ef40af1d0 legacy=0x7f4ef41e4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 shutdown_connections 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 >> 172.21.15.22:0/3910604634 conn(0x7f4ef406b660 msgr2=0x7f4ef413ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 shutdown_connections 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.958+0000 7f4ef8f8f640 1 -- 172.21.15.22:0/3910604634 wait complete. 2024-03-29T10:57:07.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:07.959+0000 7f4ef8f8f640 1 librados: shutdown 2024-03-29T10:57:07.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:14.763 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.933+0000 7f0c6691c640 10 monclient: get_monmap_and_config 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.933+0000 7f0c6691c640 10 monclient: build_initial_monmap 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.933+0000 7f0c6691c640 10 monclient: monmap: 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:14.934198+0000 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:14.934198+0000 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:14.914 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:14.915 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:14.915 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:14.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.934+0000 7f0c6691c640 1 Processor -- start 2024-03-29T10:57:14.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.934+0000 7f0c6691c640 1 -- start start 2024-03-29T10:57:14.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.934+0000 7f0c6691c640 10 monclient: init 2024-03-29T10:57:14.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.935+0000 7f0c6691c640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:14.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.935+0000 7f0c6691c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:14.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.935+0000 7f0c6691c640 10 monclient(hunting): picked mon.noname-c con 0x7f0c600b0770 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:14.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.937+0000 7f0c6691c640 10 monclient(hunting): picked mon.noname-a con 0x7f0c6014a700 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:14.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.937+0000 7f0c6691c640 10 monclient(hunting): picked mon.noname-b con 0x7f0c600ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:14.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.937+0000 7f0c6691c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600aba70 con 0x7f0c6014a700 2024-03-29T10:57:14.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.937+0000 7f0c6691c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600aecb0 con 0x7f0c600b0770 2024-03-29T10:57:14.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c6691c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c600af150 con 0x7f0c600ac5a0 2024-03-29T10:57:14.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c6691c640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:14.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c6691c640 10 monclient(hunting): authenticate will time out at 936.242139s 2024-03-29T10:57:14.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5f7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f0c6014a700 0x7f0c600abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:57767/0 (socket says 172.21.15.22:57767) 2024-03-29T10:57:14.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5f7fe640 1 -- 172.21.15.22:0/694064951 learned_addr learned my addr 172.21.15.22:0/694064951 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:14.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508484543 0 0) 0x7f0c600aba70 con 0x7f0c6014a700 2024-03-29T10:57:14.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5effd640 10 monclient(hunting): my global_id is 4332 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.938+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c54003520 con 0x7f0c6014a700 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268335161 0 0) 0x7f0c600af150 con 0x7f0c600ac5a0 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 10 monclient(hunting): my global_id is 4367 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c600aba70 con 0x7f0c600ac5a0 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2423458370 0 0) 0x7f0c600aba70 con 0x7f0c600ac5a0 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.939+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 >> v1:172.21.15.26:6789/0 conn(0x7f0c600b0770 legacy=0x7f0c60149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 >> v1:172.21.15.22:6789/0 conn(0x7f0c6014a700 legacy=0x7f0c600abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 1 monclient: found mon.noname-b 2024-03-29T10:57:14.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c600ac9d0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: _finish_auth 0 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0c48001670 con 0x7f0c600ac5a0 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: dump: 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.925 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c6691c640 5 monclient: authenticate success, global_id 4367 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c6691c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c480017d0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 1 -- 172.21.15.22:0/694064951 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0c48001ba0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c6691c640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:14.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c5effd640 10 monclient: dump: 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.942+0000 7f0c6691c640 4 set_mon_vals no callback set 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.943+0000 7f0c6691c640 10 monclient: shutdown 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.943+0000 7f0c6691c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:14.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.943+0000 7f0c6691c640 1 -- 172.21.15.22:0/694064951 >> v1:172.21.15.22:6790/0 conn(0x7f0c600ac5a0 legacy=0x7f0c60151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.945+0000 7f0c6691c640 1 -- 172.21.15.22:0/694064951 shutdown_connections 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.945+0000 7f0c6691c640 1 -- 172.21.15.22:0/694064951 >> 172.21.15.22:0/694064951 conn(0x7f0c6006b660 msgr2=0x7f0c6006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.946+0000 7f0c6691c640 1 -- 172.21.15.22:0/694064951 shutdown_connections 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.946+0000 7f0c6691c640 1 -- 172.21.15.22:0/694064951 wait complete. 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.946+0000 7f0c6691c640 10 monclient: build_initial_monmap 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.946+0000 7f0c6691c640 10 monclient: monmap: 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:14.929 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.947+0000 7f0c6691c640 1 librados: starting msgr at 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.947+0000 7f0c6691c640 1 librados: starting objecter 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.947+0000 7f0c6691c640 1 Processor -- start 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.947+0000 7f0c6691c640 1 -- start start 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.947+0000 7f0c6691c640 1 librados: setting wanted keys 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.947+0000 7f0c6691c640 1 librados: calling monclient init 2024-03-29T10:57:14.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.947+0000 7f0c6691c640 10 monclient: init 2024-03-29T10:57:14.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.948+0000 7f0c6691c640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:14.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.948+0000 7f0c6691c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:14.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.948+0000 7f0c6691c640 10 monclient(hunting): picked mon.noname-a con 0x7f0c600ac5a0 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.950+0000 7f0c6691c640 10 monclient(hunting): picked mon.noname-b con 0x7f0c600b0770 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c6691c640 10 monclient(hunting): picked mon.noname-c con 0x7f0c6014a700 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c6691c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c601239d0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c6691c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c60128200 con 0x7f0c600b0770 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c6691c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c60125e80 con 0x7f0c6014a700 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c6691c640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c6691c640 10 monclient(hunting): authenticate will time out at 936.255183s 2024-03-29T10:57:14.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5ffff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f0c600ac5a0 0x7f0c60144df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58271/0 (socket says 172.21.15.22:58271) 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5ffff640 1 -- 172.21.15.22:0/3442219817 learned_addr learned my addr 172.21.15.22:0/3442219817 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 client.?.objecter ms_handle_connect 0x7f0c6014a700 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 client.?.objecter ms_handle_connect 0x7f0c600ac5a0 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1555825472 0 0) 0x7f0c60125e80 con 0x7f0c6014a700 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): my global_id is 4370 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c40004eb0 con 0x7f0c6014a700 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929489574 0 0) 0x7f0c601239d0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:14.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 10 monclient(hunting): my global_id is 4335 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.951+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c60125e80 con 0x7f0c600ac5a0 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2789798794 0 0) 0x7f0c60125e80 con 0x7f0c600ac5a0 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c601239d0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0c58001bb0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:14.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 10 monclient(hunting): dump: 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142206116 0 0) 0x7f0c40004eb0 con 0x7f0c6014a700 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0c60125e80 con 0x7f0c6014a700 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0c48001ae0 con 0x7f0c6014a700 2024-03-29T10:57:14.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 10 monclient(hunting): dump: 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2457231500 0 0) 0x7f0c601239d0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 >> v1:172.21.15.22:6790/0 conn(0x7f0c6014a700 legacy=0x7f0c601e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:14.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.952+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 >> v1:172.21.15.26:6789/0 conn(0x7f0c600b0770 legacy=0x7f0c601dbb60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: _finish_hunting 0 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 1 monclient: found mon.a 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c60126de0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: _finish_auth 0 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: _check_auth_tickets 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 5 monclient: authenticate success, global_id 4335 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 10 monclient: _renew_subs 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c60127270 con 0x7f0c600ac5a0 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:14.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 10 monclient: _renew_subs 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c60126580 con 0x7f0c600ac5a0 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c6691c640 1 librados: init done 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0c580020c0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0c58004e20 con 0x7f0c600ac5a0 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.954+0000 7f0c5cff9640 10 monclient: dump: 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:14.943 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.955+0000 7f0c5e7fc640 4 set_mon_vals no callback set 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.956+0000 7f0c4e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.956+0000 7f0c4e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.956+0000 7f0c4e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.956+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c38005080 con 0x7f0c600ac5a0 2024-03-29T10:57:14.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.956+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f0c5801adf0 con 0x7f0c600ac5a0 2024-03-29T10:57:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.957+0000 7f0c5f7fe640 10 monclient: get_auth_request con 0x7f0c4006de40 auth_method 0 2024-03-29T10:57:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.957+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0c58084e30 con 0x7f0c600ac5a0 2024-03-29T10:57:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.957+0000 7f0c5cff9640 10 client.4335.objecter ms_dispatch 0x7f0c60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.957+0000 7f0c5cff9640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.957+0000 7f0c5cff9640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:14.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.957+0000 7f0c5cff9640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-29T10:57:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.958+0000 7f0c5cff9640 10 client.4335.objecter ms_handle_connect 0x7f0c4006de40 2024-03-29T10:57:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.964+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0c58050710 con 0x7f0c600ac5a0 2024-03-29T10:57:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.964+0000 7f0c5cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:14.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:14.964+0000 7f0c5cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:15.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.300+0000 7f0c4e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:15.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.300+0000 7f0c4e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:15.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.300+0000 7f0c4e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:57:15.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.300+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c38005330 con 0x7f0c600ac5a0 2024-03-29T10:57:15.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.301+0000 7f0c5cff9640 1 -- 172.21.15.22:0/3442219817 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0c58052640 con 0x7f0c600ac5a0 2024-03-29T10:57:15.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.301+0000 7f0c5cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:15.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.301+0000 7f0c5cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:15.282 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:15.282 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.304+0000 7f0c4e7fc640 10 librados: watch_flush enter 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.305+0000 7f0c4e7fc640 10 librados: watch_flush exit 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.305+0000 7f0c4e7fc640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.305+0000 7f0c4e7fc640 10 client.4335.objecter successfully canceled tick 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.305+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 >> v1:172.21.15.22:6816/380221644 conn(0x7f0c4006de40 legacy=0x7f0c40070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.305+0000 7f0c4e7fc640 10 monclient: shutdown 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.305+0000 7f0c4e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:15.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.305+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 >> v1:172.21.15.22:6789/0 conn(0x7f0c600ac5a0 legacy=0x7f0c60144df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:15.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.306+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 shutdown_connections 2024-03-29T10:57:15.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.306+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 >> 172.21.15.22:0/3442219817 conn(0x7f0c6006b660 msgr2=0x7f0c600b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:15.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.307+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 shutdown_connections 2024-03-29T10:57:15.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.307+0000 7f0c4e7fc640 1 -- 172.21.15.22:0/3442219817 wait complete. 2024-03-29T10:57:15.288 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:15.307+0000 7f0c4e7fc640 1 librados: shutdown 2024-03-29T10:57:15.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:22.110 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:22.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.270+0000 7fc393ef4640 10 monclient: get_monmap_and_config 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.270+0000 7fc393ef4640 10 monclient: build_initial_monmap 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.270+0000 7fc393ef4640 10 monclient: monmap: 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:22.272718+0000 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:22.272718+0000 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:22.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.271+0000 7fc393ef4640 1 Processor -- start 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.272+0000 7fc393ef4640 1 -- start start 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.272+0000 7fc393ef4640 10 monclient: init 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.272+0000 7fc393ef4640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.272+0000 7fc393ef4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.272+0000 7fc393ef4640 10 monclient(hunting): picked mon.noname-a con 0x7fc38c0b0770 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.272+0000 7fc393ef4640 10 monclient(hunting): picked mon.noname-c con 0x7fc38c14a700 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.272+0000 7fc393ef4640 10 monclient(hunting): picked mon.noname-b con 0x7fc38c0ac5a0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.273+0000 7fc393ef4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0aba70 con 0x7fc38c0b0770 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.273+0000 7fc393ef4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0aecb0 con 0x7fc38c14a700 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.273+0000 7fc393ef4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c0af150 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.273+0000 7fc393ef4640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:22.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.273+0000 7fc393ef4640 10 monclient(hunting): authenticate will time out at 943.578116s 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.273+0000 7fc392491640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc38c0ac5a0 0x7fc38c151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53004/0 (socket says 172.21.15.22:53004) 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.273+0000 7fc392491640 1 -- 172.21.15.22:0/1886719355 learned_addr learned my addr 172.21.15.22:0/1886719355 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.274+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523006076 0 0) 0x7fc38c0af150 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.274+0000 7fc390c8e640 10 monclient(hunting): _negotiate 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.274+0000 7fc390c8e640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.274+0000 7fc390c8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.274+0000 7fc390c8e640 10 monclient(hunting): my global_id is 4373 2024-03-29T10:57:22.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.274+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc374003520 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.275+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2995600257 0 0) 0x7fc374003520 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.275+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 >> v1:172.21.15.26:6789/0 conn(0x7fc38c14a700 legacy=0x7fc38c0abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:22.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.277+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 >> v1:172.21.15.22:6789/0 conn(0x7fc38c0b0770 legacy=0x7fc38c149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:22.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.279+0000 7fc390c8e640 10 monclient: _finish_hunting 0 2024-03-29T10:57:22.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.279+0000 7fc390c8e640 1 monclient: found mon.noname-b 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c0ac9d0 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: _finish_auth 0 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: _check_auth_tickets 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc393ef4640 5 monclient: authenticate success, global_id 4373 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc393ef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc384001690 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:22.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: dump: 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc393ef4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc384001860 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:22.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc393ef4640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc393ef4640 4 set_mon_vals no callback set 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 1 -- 172.21.15.22:0/1886719355 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc384001cb0 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.280+0000 7fc390c8e640 10 monclient: dump: 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:22.264 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:22.265 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:22.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.281+0000 7fc393ef4640 10 monclient: shutdown 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.281+0000 7fc393ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.281+0000 7fc393ef4640 1 -- 172.21.15.22:0/1886719355 >> v1:172.21.15.22:6790/0 conn(0x7fc38c0ac5a0 legacy=0x7fc38c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.281+0000 7fc393ef4640 1 -- 172.21.15.22:0/1886719355 shutdown_connections 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.281+0000 7fc393ef4640 1 -- 172.21.15.22:0/1886719355 >> 172.21.15.22:0/1886719355 conn(0x7fc38c06b660 msgr2=0x7fc38c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.281+0000 7fc393ef4640 1 -- 172.21.15.22:0/1886719355 shutdown_connections 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.281+0000 7fc393ef4640 1 -- 172.21.15.22:0/1886719355 wait complete. 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.282+0000 7fc393ef4640 10 monclient: build_initial_monmap 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.282+0000 7fc393ef4640 10 monclient: monmap: 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:22.267 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.282+0000 7fc393ef4640 1 librados: starting msgr at 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.282+0000 7fc393ef4640 1 librados: starting objecter 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.283+0000 7fc393ef4640 1 Processor -- start 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.283+0000 7fc393ef4640 1 -- start start 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.283+0000 7fc393ef4640 1 librados: setting wanted keys 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.283+0000 7fc393ef4640 1 librados: calling monclient init 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.283+0000 7fc393ef4640 10 monclient: init 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.283+0000 7fc393ef4640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.283+0000 7fc393ef4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:22.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 10 monclient(hunting): picked mon.noname-a con 0x7fc38c0ac5a0 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 10 monclient(hunting): picked mon.noname-c con 0x7fc38c0b0770 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 10 monclient(hunting): picked mon.noname-b con 0x7fc38c14a700 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c128190 con 0x7fc38c0ac5a0 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c1dea90 con 0x7fc38c14a700 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc38c1df910 con 0x7fc38c0b0770 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc393ef4640 10 monclient(hunting): authenticate will time out at 943.589552s 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.284+0000 7fc39148f640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc38c0b0770 0x7fc38c1df200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53166/0 (socket says 172.21.15.22:53166) 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc39148f640 1 -- 172.21.15.22:0/1749601600 learned_addr learned my addr 172.21.15.22:0/1749601600 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 client.?.objecter ms_handle_connect 0x7fc38c0b0770 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154289510 0 0) 0x7fc38c1df910 con 0x7fc38c0b0770 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:22.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): my global_id is 4376 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc378004eb0 con 0x7fc38c0b0770 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4035827144 0 0) 0x7fc378004eb0 con 0x7fc38c0b0770 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc378005300 con 0x7fc38c0b0770 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc380001790 con 0x7fc38c0b0770 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): dump: 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.270 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.285+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 284750936 0 0) 0x7fc378005300 con 0x7fc38c0b0770 2024-03-29T10:57:22.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.286+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 >> v1:172.21.15.26:6789/0 conn(0x7fc38c14a700 legacy=0x7fc38c1450b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:22.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.289+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 >> v1:172.21.15.22:6789/0 conn(0x7fc38c0ac5a0 legacy=0x7fc38c126d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:22.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 1 monclient: found mon.c 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38c1283e0 con 0x7fc38c0b0770 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc38a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc393ef4640 5 monclient: authenticate success, global_id 4376 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc393ef4640 10 monclient: _renew_subs 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc393ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.290+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc38c1dece0 con 0x7fc38c0b0770 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc393ef4640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc393ef4640 10 monclient: _renew_subs 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc393ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:22.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc38c145ab0 con 0x7fc38c0b0770 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc393ef4640 1 librados: init done 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc380001d60 con 0x7fc38c0b0770 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc38a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc380001790 con 0x7fc38c0b0770 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc38a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc38a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc38a7fc640 10 monclient: dump: 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:22.274 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:22.275 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:22.275 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:22.275 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:22.275 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:22.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.291+0000 7fc38bfff640 4 set_mon_vals no callback set 2024-03-29T10:57:22.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.292+0000 7fc393ef4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:22.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.292+0000 7fc393ef4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:22.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.292+0000 7fc393ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:22.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.292+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc38c1def20 con 0x7fc38c0b0770 2024-03-29T10:57:22.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.295+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fc38001a6d0 con 0x7fc38c0b0770 2024-03-29T10:57:22.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.296+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fc380084720 con 0x7fc38c0b0770 2024-03-29T10:57:22.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.296+0000 7fc38a7fc640 10 client.4376.objecter ms_dispatch 0x7fc38c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:22.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.296+0000 7fc38a7fc640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:22.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.296+0000 7fc38a7fc640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:22.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.296+0000 7fc38a7fc640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-29T10:57:22.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.296+0000 7fc391c90640 10 monclient: get_auth_request con 0x7fc37806e090 auth_method 0 2024-03-29T10:57:22.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.297+0000 7fc38a7fc640 10 client.4376.objecter ms_handle_connect 0x7fc37806e090 2024-03-29T10:57:22.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.303+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc38c1def20 con 0x7fc38c0b0770 2024-03-29T10:57:22.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.303+0000 7fc38a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:22.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.303+0000 7fc38a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:22.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.623+0000 7fc393ef4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:22.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.623+0000 7fc393ef4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:22.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.623+0000 7fc393ef4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:22.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.623+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc38c0adf60 con 0x7fc38c0b0770 2024-03-29T10:57:22.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.624+0000 7fc38a7fc640 1 -- 172.21.15.22:0/1749601600 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fc38004fe30 con 0x7fc38c0b0770 2024-03-29T10:57:22.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.624+0000 7fc38a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:22.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.624+0000 7fc38a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:22.606 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:22.606 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:22.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.626+0000 7fc393ef4640 10 librados: watch_flush enter 2024-03-29T10:57:22.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.629+0000 7fc393ef4640 10 librados: watch_flush exit 2024-03-29T10:57:22.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.629+0000 7fc393ef4640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-29T10:57:22.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.629+0000 7fc393ef4640 10 client.4376.objecter successfully canceled tick 2024-03-29T10:57:22.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.629+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 >> v1:172.21.15.22:6816/380221644 conn(0x7fc37806e090 legacy=0x7fc378070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.629+0000 7fc393ef4640 10 monclient: shutdown 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.629+0000 7fc393ef4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.629+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 >> v1:172.21.15.22:6790/0 conn(0x7fc38c0b0770 legacy=0x7fc38c1df200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.630+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 shutdown_connections 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.630+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 >> 172.21.15.22:0/1749601600 conn(0x7fc38c06b660 msgr2=0x7fc38c0abe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.630+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 shutdown_connections 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.630+0000 7fc393ef4640 1 -- 172.21.15.22:0/1749601600 wait complete. 2024-03-29T10:57:22.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:22.630+0000 7fc393ef4640 1 librados: shutdown 2024-03-29T10:57:22.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:29.432 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.587+0000 7f2dcf59e640 10 monclient: get_monmap_and_config 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.587+0000 7f2dcf59e640 10 monclient: build_initial_monmap 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.587+0000 7f2dcf59e640 10 monclient: monmap: 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:29.589663+0000 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:29.589663+0000 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:29.571 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:29.572 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:29.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.589+0000 7f2dcf59e640 1 Processor -- start 2024-03-29T10:57:29.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.589+0000 7f2dcf59e640 1 -- start start 2024-03-29T10:57:29.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.589+0000 7f2dcf59e640 10 monclient: init 2024-03-29T10:57:29.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.591+0000 7f2dcf59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:29.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.591+0000 7f2dcf59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc8034850 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc80c9550 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcdd9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2dc80c9550 0x7f2dc80cbea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59902/0 (socket says 172.21.15.22:59902) 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc802aac0 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc802c440 con 0x7f2dc8034850 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc802aed0 con 0x7f2dc802aac0 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc802b190 con 0x7f2dc80c9550 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:29.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcf59e640 10 monclient(hunting): authenticate will time out at 950.897970s 2024-03-29T10:57:29.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.592+0000 7f2dcdd9b640 1 -- 172.21.15.22:0/130470620 learned_addr learned my addr 172.21.15.22:0/130470620 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:29.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.593+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664375061 0 0) 0x7f2dc802b190 con 0x7f2dc80c9550 2024-03-29T10:57:29.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.593+0000 7f2dcd59a640 10 monclient(hunting): _negotiate 2024-03-29T10:57:29.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.593+0000 7f2dcd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:29.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.593+0000 7f2dcd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:29.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.593+0000 7f2dcd59a640 10 monclient(hunting): my global_id is 4379 2024-03-29T10:57:29.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.593+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db8003520 con 0x7f2dc80c9550 2024-03-29T10:57:29.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.595+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4180395998 0 0) 0x7f2db8003520 con 0x7f2dc80c9550 2024-03-29T10:57:29.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.595+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 >> v1:172.21.15.26:6789/0 conn(0x7f2dc802aac0 legacy=0x7f2dc80cc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:57:29.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.595+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 >> v1:172.21.15.22:6789/0 conn(0x7f2dc8034850 legacy=0x7f2dc80c8e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:29.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: _finish_hunting 0 2024-03-29T10:57:29.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 1 monclient: found mon.noname-b 2024-03-29T10:57:29.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:29.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:57:29.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8033060 con 0x7f2dc80c9550 2024-03-29T10:57:29.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: _finish_auth 0 2024-03-29T10:57:29.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: _check_auth_tickets 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcf59e640 5 monclient: authenticate success, global_id 4379 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2dc0001670 con 0x7f2dc80c9550 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: dump: 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:29.582 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dc00018d0 con 0x7f2dc80c9550 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcf59e640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcf59e640 4 set_mon_vals no callback set 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 1 -- 172.21.15.22:0/130470620 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2dc0001d20 con 0x7f2dc80c9550 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.598+0000 7f2dcd59a640 10 monclient: dump: 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:29.583 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.599+0000 7f2dcf59e640 10 monclient: shutdown 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.599+0000 7f2dcf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.599+0000 7f2dcf59e640 1 -- 172.21.15.22:0/130470620 >> v1:172.21.15.22:6790/0 conn(0x7f2dc80c9550 legacy=0x7f2dc80cbea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.599+0000 7f2dcf59e640 1 -- 172.21.15.22:0/130470620 shutdown_connections 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.599+0000 7f2dcf59e640 1 -- 172.21.15.22:0/130470620 >> 172.21.15.22:0/130470620 conn(0x7f2dc80058f0 msgr2=0x7f2dc8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.599+0000 7f2dcf59e640 1 -- 172.21.15.22:0/130470620 shutdown_connections 2024-03-29T10:57:29.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.599+0000 7f2dcf59e640 1 -- 172.21.15.22:0/130470620 wait complete. 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.600+0000 7f2dcf59e640 10 monclient: build_initial_monmap 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.600+0000 7f2dcf59e640 10 monclient: monmap: 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:29.585 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.601+0000 7f2dcf59e640 1 librados: starting msgr at 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.601+0000 7f2dcf59e640 1 librados: starting objecter 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.601+0000 7f2dcf59e640 1 Processor -- start 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.603+0000 7f2dcf59e640 1 -- start start 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.603+0000 7f2dcf59e640 1 librados: setting wanted keys 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.603+0000 7f2dcf59e640 1 librados: calling monclient init 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.603+0000 7f2dcf59e640 10 monclient: init 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.603+0000 7f2dcf59e640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.603+0000 7f2dcf59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f2dc802aac0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:29.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dce59c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2dc802aac0 0x7f2dc8164680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60348/0 (socket says 172.21.15.22:60348) 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f2dc8034850 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dce59c640 1 -- 172.21.15.22:0/2756534151 learned_addr learned my addr 172.21.15.22:0/2756534151 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f2dc80c9550 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc802ebd0 con 0x7f2dc8034850 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc802eeb0 con 0x7f2dc80c9550 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dc8164160 con 0x7f2dc802aac0 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dcf59e640 10 monclient(hunting): authenticate will time out at 950.909300s 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dbeffd640 10 client.?.objecter ms_handle_connect 0x7f2dc802aac0 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.604+0000 7f2dbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216667444 0 0) 0x7f2dc8164160 con 0x7f2dc802aac0 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:29.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): my global_id is 4382 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2db0004eb0 con 0x7f2dc802aac0 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1901604110 0 0) 0x7f2db0004eb0 con 0x7f2dc802aac0 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dc8164160 con 0x7f2dc802aac0 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2dd004fdd0 con 0x7f2dc802aac0 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): dump: 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:29.590 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:29.591 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:29.591 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:29.591 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:29.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:29.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 587508398 0 0) 0x7f2dc8164160 con 0x7f2dc802aac0 2024-03-29T10:57:29.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.605+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 >> v1:172.21.15.26:6789/0 conn(0x7f2dc80c9550 legacy=0x7f2dc816c120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:29.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.608+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 >> v1:172.21.15.22:6789/0 conn(0x7f2dc8034850 legacy=0x7f2dc80b95f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:57:29.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:29.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 1 monclient: found mon.c 2024-03-29T10:57:29.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dc8163d80 con 0x7f2dc802aac0 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.611+0000 7f2dbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.612+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dd004f400 con 0x7f2dc802aac0 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.612+0000 7f2dbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.612+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2dd00008d0 con 0x7f2dc802aac0 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.612+0000 7f2dbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.612+0000 7f2dbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.612+0000 7f2dbeffd640 10 monclient: dump: 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:29.594 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.595 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:29.595 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:29.595 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:29.595 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:29.595 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:29.595 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:29.595 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 5 monclient: authenticate success, global_id 4382 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 10 monclient: _renew_subs 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dc80b9d00 con 0x7f2dc802aac0 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 10 monclient: _renew_subs 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dc802db90 con 0x7f2dc802aac0 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.613+0000 7f2dcf59e640 1 librados: init done 2024-03-29T10:57:29.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.614+0000 7f2dbffff640 4 set_mon_vals no callback set 2024-03-29T10:57:29.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.615+0000 7f2dcf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.615+0000 7f2dcf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.615+0000 7f2dcf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.615+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db4005080 con 0x7f2dc802aac0 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.615+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f2dd0067bf0 con 0x7f2dc802aac0 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.616+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f2dd00ff750 con 0x7f2dc802aac0 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.616+0000 7f2dbeffd640 10 client.4382.objecter ms_dispatch 0x7f2dd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.616+0000 7f2dbeffd640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.616+0000 7f2dbeffd640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.616+0000 7f2dbeffd640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T10:57:29.598 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.617+0000 7f2dcdd9b640 10 monclient: get_auth_request con 0x7f2db006db70 auth_method 0 2024-03-29T10:57:29.599 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.618+0000 7f2dbeffd640 10 client.4382.objecter ms_handle_connect 0x7f2db006db70 2024-03-29T10:57:29.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.626+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dd00cb090 con 0x7f2dc802aac0 2024-03-29T10:57:29.607 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.626+0000 7f2dbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:29.608 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.626+0000 7f2dbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:29.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.934+0000 7f2dcf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:29.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.934+0000 7f2dcf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:29.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.934+0000 7f2dcf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:29.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.934+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2db4005330 con 0x7f2dc802aac0 2024-03-29T10:57:29.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.937+0000 7f2dbeffd640 1 -- 172.21.15.22:0/2756534151 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f2dd00ccfc0 con 0x7f2dc802aac0 2024-03-29T10:57:29.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.937+0000 7f2dbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:29.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.937+0000 7f2dbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:29.919 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:29.919 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:29.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.940+0000 7f2dcf59e640 10 librados: watch_flush enter 2024-03-29T10:57:29.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.943+0000 7f2dcf59e640 10 librados: watch_flush exit 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.943+0000 7f2dcf59e640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.943+0000 7f2dcf59e640 10 client.4382.objecter successfully canceled tick 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.943+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 >> v1:172.21.15.22:6816/380221644 conn(0x7f2db006db70 legacy=0x7f2db0070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.943+0000 7f2dcf59e640 10 monclient: shutdown 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.943+0000 7f2dcf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.943+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 >> v1:172.21.15.22:6790/0 conn(0x7f2dc802aac0 legacy=0x7f2dc8164680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.944+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 shutdown_connections 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.944+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 >> 172.21.15.22:0/2756534151 conn(0x7f2dc80058f0 msgr2=0x7f2dc80043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.944+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 shutdown_connections 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.944+0000 7f2dcf59e640 1 -- 172.21.15.22:0/2756534151 wait complete. 2024-03-29T10:57:29.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:29.944+0000 7f2dcf59e640 1 librados: shutdown 2024-03-29T10:57:29.943 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:36.750 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:36.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.922+0000 7f41a0ffa640 10 monclient: get_monmap_and_config 2024-03-29T10:57:36.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.922+0000 7f41a0ffa640 10 monclient: build_initial_monmap 2024-03-29T10:57:36.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.922+0000 7f41a0ffa640 10 monclient: monmap: 2024-03-29T10:57:36.903 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:36.903 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:36.903 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:36.923482+0000 2024-03-29T10:57:36.904 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:36.923482+0000 2024-03-29T10:57:36.904 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:36.904 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:36.904 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:36.904 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:36.904 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:36.904 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:36.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.926+0000 7f41a0ffa640 1 Processor -- start 2024-03-29T10:57:36.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.926+0000 7f41a0ffa640 1 -- start start 2024-03-29T10:57:36.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.926+0000 7f41a0ffa640 10 monclient: init 2024-03-29T10:57:36.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 10 monclient(hunting): picked mon.noname-a con 0x7f419c0b2800 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 10 monclient(hunting): picked mon.noname-c con 0x7f419c0a9c00 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 10 monclient(hunting): picked mon.noname-b con 0x7f419c0b38c0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c141eb0 con 0x7f419c0b2800 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c0b22d0 con 0x7f419c0a9c00 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c142d30 con 0x7f419c0b38c0 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:36.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.927+0000 7f41a0ffa640 10 monclient(hunting): authenticate will time out at 958.231737s 2024-03-29T10:57:36.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.928+0000 7f419bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f419c0b38c0 0x7f419c14d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41510/0 (socket says 172.21.15.22:41510) 2024-03-29T10:57:36.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.928+0000 7f419bfff640 1 -- 172.21.15.22:0/1202342876 learned_addr learned my addr 172.21.15.22:0/1202342876 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:36.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344317899 0 0) 0x7f419c142d30 con 0x7f419c0b38c0 2024-03-29T10:57:36.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:57:36.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:36.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:36.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 10 monclient(hunting): my global_id is 4385 2024-03-29T10:57:36.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4184003520 con 0x7f419c0b38c0 2024-03-29T10:57:36.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3605524651 0 0) 0x7f4184003520 con 0x7f419c0b38c0 2024-03-29T10:57:36.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.929+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 >> v1:172.21.15.26:6789/0 conn(0x7f419c0a9c00 legacy=0x7f419c13bb60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:36.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.930+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 >> v1:172.21.15.22:6789/0 conn(0x7f419c0b2800 legacy=0x7f419c142620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:36.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:57:36.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c0aaf90 con 0x7f419c0b38c0 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f41940019f0 con 0x7f419c0b38c0 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.931+0000 7f419a7fc640 10 monclient: dump: 2024-03-29T10:57:36.916 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f41a0ffa640 5 monclient: authenticate success, global_id 4385 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f41a0ffa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4194001c50 con 0x7f419c0b38c0 2024-03-29T10:57:36.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f419a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f419a7fc640 1 -- 172.21.15.22:0/1202342876 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4194002020 con 0x7f419c0b38c0 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f419a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f419a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f419a7fc640 10 monclient: dump: 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:36.918 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f41a0ffa640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.932+0000 7f41a0ffa640 4 set_mon_vals no callback set 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.933+0000 7f41a0ffa640 10 monclient: shutdown 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.933+0000 7f41a0ffa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.933+0000 7f41a0ffa640 1 -- 172.21.15.22:0/1202342876 >> v1:172.21.15.22:6790/0 conn(0x7f419c0b38c0 legacy=0x7f419c14d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.933+0000 7f41a0ffa640 1 -- 172.21.15.22:0/1202342876 shutdown_connections 2024-03-29T10:57:36.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.933+0000 7f41a0ffa640 1 -- 172.21.15.22:0/1202342876 >> 172.21.15.22:0/1202342876 conn(0x7f419c066a00 msgr2=0x7f419c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:36.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.936+0000 7f41a0ffa640 1 -- 172.21.15.22:0/1202342876 shutdown_connections 2024-03-29T10:57:36.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.938+0000 7f41a0ffa640 1 -- 172.21.15.22:0/1202342876 wait complete. 2024-03-29T10:57:36.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.941+0000 7f41a0ffa640 10 monclient: build_initial_monmap 2024-03-29T10:57:36.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.941+0000 7f41a0ffa640 10 monclient: monmap: 2024-03-29T10:57:36.928 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:36.928 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:36.928 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:36.928 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:36.929 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:36.929 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:36.929 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:36.929 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:36.929 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:36.929 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:36.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.941+0000 7f41a0ffa640 1 librados: starting msgr at 2024-03-29T10:57:36.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.941+0000 7f41a0ffa640 1 librados: starting objecter 2024-03-29T10:57:36.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.941+0000 7f41a0ffa640 1 Processor -- start 2024-03-29T10:57:36.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.942+0000 7f41a0ffa640 1 -- start start 2024-03-29T10:57:36.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.942+0000 7f41a0ffa640 1 librados: setting wanted keys 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.942+0000 7f41a0ffa640 1 librados: calling monclient init 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.942+0000 7f41a0ffa640 10 monclient: init 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.942+0000 7f41a0ffa640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.942+0000 7f41a0ffa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.942+0000 7f41a0ffa640 10 monclient(hunting): picked mon.noname-a con 0x7f419c0a9c00 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f41a0ffa640 10 monclient(hunting): picked mon.noname-c con 0x7f419c0b2800 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f419affd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f419c0b2800 0x7f419c13ba80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41676/0 (socket says 172.21.15.22:41676) 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f419affd640 1 -- 172.21.15.22:0/4122407598 learned_addr learned my addr 172.21.15.22:0/4122407598 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f41a0ffa640 10 monclient(hunting): picked mon.noname-b con 0x7f419c0b38c0 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:36.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c06fe80 con 0x7f419c0a9c00 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c1e3980 con 0x7f419c0b38c0 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f419c1e45b0 con 0x7f419c0b2800 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f41a0ffa640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f41a0ffa640 10 monclient(hunting): authenticate will time out at 958.247399s 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 10 client.?.objecter ms_handle_connect 0x7f419c0b2800 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38370171 0 0) 0x7f419c1e45b0 con 0x7f419c0b2800 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 10 monclient(hunting): my global_id is 4388 2024-03-29T10:57:36.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.943+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4188004eb0 con 0x7f419c0b2800 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3643746542 0 0) 0x7f4188004eb0 con 0x7f419c0b2800 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f419c1e45b0 con 0x7f419c0b2800 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f418c001870 con 0x7f419c0b2800 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 10 monclient(hunting): dump: 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.933 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 520147485 0 0) 0x7f419c1e45b0 con 0x7f419c0b2800 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 >> v1:172.21.15.26:6789/0 conn(0x7f419c0b38c0 legacy=0x7f419c1e3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.946+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 >> v1:172.21.15.22:6789/0 conn(0x7f419c0a9c00 legacy=0x7f419c06e760 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 1 monclient: found mon.c 2024-03-29T10:57:36.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:36.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:36.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f419c1e4740 con 0x7f419c0b2800 2024-03-29T10:57:36.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 10 monclient: _finish_auth 0 2024-03-29T10:57:36.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:36.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f4183fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:36.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 5 monclient: authenticate success, global_id 4388 2024-03-29T10:57:36.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 10 monclient: _renew_subs 2024-03-29T10:57:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f419c138f90 con 0x7f419c0b2800 2024-03-29T10:57:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 10 monclient: _renew_subs 2024-03-29T10:57:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f419c0718a0 con 0x7f419c0b2800 2024-03-29T10:57:36.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.947+0000 7f41a0ffa640 1 librados: init done 2024-03-29T10:57:36.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.949+0000 7f41a0ffa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:36.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.950+0000 7f41a0ffa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:36.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.950+0000 7f41a0ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:36.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.950+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f419c1e3bc0 con 0x7f419c0b2800 2024-03-29T10:57:36.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.951+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f418c0019d0 con 0x7f419c0b2800 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.951+0000 7f4183fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.951+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f418c004a70 con 0x7f419c0b2800 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.951+0000 7f4183fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.951+0000 7f4183fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.951+0000 7f4183fff640 10 monclient: dump: 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:36.940 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:36.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.954+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f418c01ab00 con 0x7f419c0b2800 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.954+0000 7f41997fa640 4 set_mon_vals no callback set 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.955+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f418c084ac0 con 0x7f419c0b2800 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.955+0000 7f4183fff640 10 client.4388.objecter ms_dispatch 0x7f419c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.955+0000 7f4183fff640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.955+0000 7f4183fff640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.955+0000 7f4183fff640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.955+0000 7f419b7fe640 10 monclient: get_auth_request con 0x7f418806dc90 auth_method 0 2024-03-29T10:57:36.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.955+0000 7f4183fff640 10 client.4388.objecter ms_handle_connect 0x7f418806dc90 2024-03-29T10:57:36.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.963+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f419c1e3bc0 con 0x7f419c0b2800 2024-03-29T10:57:36.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.963+0000 7f4183fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:36.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:36.963+0000 7f4183fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:37.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.297+0000 7f41a0ffa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:37.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.297+0000 7f41a0ffa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:37.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.297+0000 7f41a0ffa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:37.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.297+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f419c1e3bc0 con 0x7f419c0b2800 2024-03-29T10:57:37.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.299+0000 7f4183fff640 1 -- 172.21.15.22:0/4122407598 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f419c1e3bc0 con 0x7f419c0b2800 2024-03-29T10:57:37.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.299+0000 7f4183fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:37.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.299+0000 7f4183fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:37.280 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:37.280 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:37.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 10 librados: watch_flush enter 2024-03-29T10:57:37.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 10 librados: watch_flush exit 2024-03-29T10:57:37.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T10:57:37.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 10 client.4388.objecter successfully canceled tick 2024-03-29T10:57:37.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 >> v1:172.21.15.22:6816/380221644 conn(0x7f418806dc90 legacy=0x7f4188070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:37.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 10 monclient: shutdown 2024-03-29T10:57:37.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:37.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.303+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 >> v1:172.21.15.22:6790/0 conn(0x7f419c0b2800 legacy=0x7f419c13ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:37.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.306+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 shutdown_connections 2024-03-29T10:57:37.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.306+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 >> 172.21.15.22:0/4122407598 conn(0x7f419c066a00 msgr2=0x7f419c142620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:37.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.306+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 shutdown_connections 2024-03-29T10:57:37.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.306+0000 7f41a0ffa640 1 -- 172.21.15.22:0/4122407598 wait complete. 2024-03-29T10:57:37.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:37.306+0000 7f41a0ffa640 1 librados: shutdown 2024-03-29T10:57:37.305 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:44.113 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:44.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.291+0000 7f42bf79a640 10 monclient: get_monmap_and_config 2024-03-29T10:57:44.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.291+0000 7f42bf79a640 10 monclient: build_initial_monmap 2024-03-29T10:57:44.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.291+0000 7f42bf79a640 10 monclient: monmap: 2024-03-29T10:57:44.273 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:44.273 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:44.273 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:44.292681+0000 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:44.292681+0000 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.292+0000 7f42bf79a640 1 Processor -- start 2024-03-29T10:57:44.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.293+0000 7f42bf79a640 1 -- start start 2024-03-29T10:57:44.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.293+0000 7f42bf79a640 10 monclient: init 2024-03-29T10:57:44.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 10 monclient(hunting): picked mon.noname-a con 0x7f42b80b0500 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 10 monclient(hunting): picked mon.noname-b con 0x7f42b80b08e0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 10 monclient(hunting): picked mon.noname-c con 0x7f42b80aeca0 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b8140620 con 0x7f42b80b0500 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b813daf0 con 0x7f42b80aeca0 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b80ae180 con 0x7f42b80b08e0 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:44.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bf79a640 10 monclient(hunting): authenticate will time out at 965.598250s 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bd536640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f42b80b0500 0x7f42b813ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:58390/0 (socket says 172.21.15.22:58390) 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42bd536640 1 -- 172.21.15.22:0/1181552975 learned_addr learned my addr 172.21.15.22:0/1181552975 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.294+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923973120 0 0) 0x7f42b813daf0 con 0x7f42b80aeca0 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): my global_id is 4330 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a4003520 con 0x7f42b80aeca0 2024-03-29T10:57:44.280 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291490091 0 0) 0x7f42b8140620 con 0x7f42b80b0500 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): my global_id is 4338 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b813daf0 con 0x7f42b80b0500 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1927916672 0 0) 0x7f42b80ae180 con 0x7f42b80b08e0 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:44.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient(hunting): my global_id is 4391 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b8140620 con 0x7f42b80b08e0 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197366275 0 0) 0x7f42a4003520 con 0x7f42b80aeca0 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 >> v1:172.21.15.22:6790/0 conn(0x7f42b80b08e0 legacy=0x7f42b813d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 >> v1:172.21.15.22:6789/0 conn(0x7f42b80b0500 legacy=0x7f42b813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 monclient: found mon.noname-c 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b80accf0 con 0x7f42b80aeca0 2024-03-29T10:57:44.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient: _finish_auth 0 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f42b0001a10 con 0x7f42b80aeca0 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.295+0000 7f42affff640 10 monclient: dump: 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:44.283 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42bf79a640 5 monclient: authenticate success, global_id 4330 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42bf79a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42b0001b70 con 0x7f42b80aeca0 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42affff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42affff640 1 -- 172.21.15.22:0/1181552975 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f42b0001f80 con 0x7f42b80aeca0 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42affff640 10 monclient: dump: 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.284 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42bf79a640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.296+0000 7f42bf79a640 4 set_mon_vals no callback set 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.297+0000 7f42bf79a640 10 monclient: shutdown 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.297+0000 7f42bf79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.297+0000 7f42bf79a640 1 -- 172.21.15.22:0/1181552975 >> v1:172.21.15.26:6789/0 conn(0x7f42b80aeca0 legacy=0x7f42b814b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.298+0000 7f42bf79a640 1 -- 172.21.15.22:0/1181552975 shutdown_connections 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.298+0000 7f42bf79a640 1 -- 172.21.15.22:0/1181552975 >> 172.21.15.22:0/1181552975 conn(0x7f42b806b660 msgr2=0x7f42b806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.298+0000 7f42bf79a640 1 -- 172.21.15.22:0/1181552975 shutdown_connections 2024-03-29T10:57:44.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.298+0000 7f42bf79a640 1 -- 172.21.15.22:0/1181552975 wait complete. 2024-03-29T10:57:44.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.299+0000 7f42bf79a640 10 monclient: build_initial_monmap 2024-03-29T10:57:44.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.299+0000 7f42bf79a640 10 monclient: monmap: 2024-03-29T10:57:44.293 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:44.293 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:44.293 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.300+0000 7f42bf79a640 1 librados: starting msgr at 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.300+0000 7f42bf79a640 1 librados: starting objecter 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.300+0000 7f42bf79a640 1 Processor -- start 2024-03-29T10:57:44.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.300+0000 7f42bf79a640 1 -- start start 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.300+0000 7f42bf79a640 1 librados: setting wanted keys 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.300+0000 7f42bf79a640 1 librados: calling monclient init 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.300+0000 7f42bf79a640 10 monclient: init 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.301+0000 7f42bf79a640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.301+0000 7f42bf79a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.301+0000 7f42bf79a640 10 monclient(hunting): picked mon.noname-c con 0x7f42b80aeca0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bf79a640 10 monclient(hunting): picked mon.noname-a con 0x7f42b80b0500 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bf79a640 10 monclient(hunting): picked mon.noname-b con 0x7f42b811b1c0 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bd536640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f42b80aeca0 0x7f42b811bef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42292/0 (socket says 172.21.15.22:42292) 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bd536640 1 -- 172.21.15.22:0/3981787730 learned_addr learned my addr 172.21.15.22:0/3981787730 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b811d600 con 0x7f42b80b0500 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b811c730 con 0x7f42b811b1c0 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b813cfc0 con 0x7f42b80aeca0 2024-03-29T10:57:44.295 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bf79a640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42bf79a640 10 monclient(hunting): authenticate will time out at 965.606769s 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42adffb640 10 client.?.objecter ms_handle_connect 0x7f42b80b0500 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42adffb640 10 client.?.objecter ms_handle_connect 0x7f42b811b1c0 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.302+0000 7f42adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 client.?.objecter ms_handle_connect 0x7f42b80aeca0 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3281437807 0 0) 0x7f42b813cfc0 con 0x7f42b80aeca0 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): my global_id is 4394 2024-03-29T10:57:44.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c004eb0 con 0x7f42b80aeca0 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427460649 0 0) 0x7f42b811d600 con 0x7f42b80b0500 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): my global_id is 4341 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42b813cfc0 con 0x7f42b80b0500 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2705292775 0 0) 0x7f429c004eb0 con 0x7f42b80aeca0 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b811d600 con 0x7f42b80aeca0 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f42b40019f0 con 0x7f42b80aeca0 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.303+0000 7f42adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:44.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): dump: 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385933452 0 0) 0x7f42b811c730 con 0x7f42b811b1c0 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:44.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): my global_id is 4333 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f429c004eb0 con 0x7f42b811b1c0 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619149142 0 0) 0x7f429c004eb0 con 0x7f42b811b1c0 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42b811c730 con 0x7f42b811b1c0 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f42b0001ce0 con 0x7f42b811b1c0 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient(hunting): dump: 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.299 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 708404317 0 0) 0x7f42b813cfc0 con 0x7f42b80b0500 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f429c004eb0 con 0x7f42b80b0500 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4077489241 0 0) 0x7f42b811d600 con 0x7f42b80aeca0 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 >> v1:172.21.15.26:6789/0 conn(0x7f42b811b1c0 legacy=0x7f42b81dc400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 >> v1:172.21.15.22:6789/0 conn(0x7f42b80b0500 legacy=0x7f42b81dbcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 monclient: found mon.c 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:44.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.304+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b813d150 con 0x7f42b80aeca0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 5 monclient: authenticate success, global_id 4394 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 10 monclient: _renew_subs 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b81e74d0 con 0x7f42b80aeca0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 10 monclient: _renew_subs 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b81e7840 con 0x7f42b80aeca0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42bf79a640 1 librados: init done 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42b4001b50 con 0x7f42b80aeca0 2024-03-29T10:57:44.301 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f42b4004a90 con 0x7f42b80aeca0 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42af7fe640 4 set_mon_vals no callback set 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.305+0000 7f42adffb640 10 monclient: dump: 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:44.302 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.307+0000 7f42bf79a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.307+0000 7f42bf79a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.307+0000 7f42bf79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.307+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4298005080 con 0x7f42b80aeca0 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.308+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f42b401aa60 con 0x7f42b80aeca0 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.309+0000 7f42bcd35640 10 monclient: get_auth_request con 0x7f429c06e050 auth_method 0 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.309+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f42b4084ac0 con 0x7f42b80aeca0 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.309+0000 7f42adffb640 10 client.4394.objecter ms_dispatch 0x7f42b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.309+0000 7f42adffb640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.309+0000 7f42adffb640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.309+0000 7f42adffb640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-29T10:57:44.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.309+0000 7f42adffb640 10 client.4394.objecter ms_handle_connect 0x7f429c06e050 2024-03-29T10:57:44.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.318+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f42b40501d0 con 0x7f42b80aeca0 2024-03-29T10:57:44.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.318+0000 7f42adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:44.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.318+0000 7f42adffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:44.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.700+0000 7f42bf79a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:44.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.700+0000 7f42bf79a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:44.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.700+0000 7f42bf79a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:44.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.700+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4298005330 con 0x7f42b80aeca0 2024-03-29T10:57:44.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.701+0000 7f42adffb640 1 -- 172.21.15.22:0/3981787730 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f42b40522d0 con 0x7f42b80aeca0 2024-03-29T10:57:44.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.701+0000 7f42adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:44.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.701+0000 7f42adffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:44.682 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:44.682 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 10 librados: watch_flush enter 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 10 librados: watch_flush exit 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 10 client.4394.objecter successfully canceled tick 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 >> v1:172.21.15.22:6816/380221644 conn(0x7f429c06e050 legacy=0x7f429c070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 10 monclient: shutdown 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 >> v1:172.21.15.22:6790/0 conn(0x7f42b80aeca0 legacy=0x7f42b811bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 shutdown_connections 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.705+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 >> 172.21.15.22:0/3981787730 conn(0x7f42b806b660 msgr2=0x7f42b80b2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:44.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.706+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 shutdown_connections 2024-03-29T10:57:44.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.706+0000 7f42bf79a640 1 -- 172.21.15.22:0/3981787730 wait complete. 2024-03-29T10:57:44.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:44.706+0000 7f42bf79a640 1 librados: shutdown 2024-03-29T10:57:44.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:51.508 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:51.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.684+0000 7ffb7e058640 10 monclient: get_monmap_and_config 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.684+0000 7ffb7e058640 10 monclient: build_initial_monmap 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.684+0000 7ffb7e058640 10 monclient: monmap: 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:51.685391+0000 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:51.685391+0000 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:51.665 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:51.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.685+0000 7ffb7e058640 1 Processor -- start 2024-03-29T10:57:51.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.686+0000 7ffb7e058640 1 -- start start 2024-03-29T10:57:51.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.686+0000 7ffb7e058640 10 monclient: init 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 10 monclient(hunting): picked mon.noname-c con 0x7ffb780b2800 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 10 monclient(hunting): picked mon.noname-b con 0x7ffb780a9c00 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7d056640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffb780b2800 0x7ffb78142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:52188/0 (socket says 172.21.15.22:52188) 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 10 monclient(hunting): picked mon.noname-a con 0x7ffb780b38c0 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7d857640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ffb780b38c0 0x7ffb7814d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:33038/0 (socket says 172.21.15.22:33038) 2024-03-29T10:57:51.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7d857640 1 -- 172.21.15.22:0/3398978950 learned_addr learned my addr 172.21.15.22:0/3398978950 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:51.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb78141eb0 con 0x7ffb780b38c0 2024-03-29T10:57:51.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780b22d0 con 0x7ffb780b2800 2024-03-29T10:57:51.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.687+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb78142d30 con 0x7ffb780a9c00 2024-03-29T10:57:51.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb7e058640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:51.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb7e058640 10 monclient(hunting): authenticate will time out at 972.992206s 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 24683603 0 0) 0x7ffb78141eb0 con 0x7ffb780b38c0 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): my global_id is 4344 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64003520 con 0x7ffb780b38c0 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075306129 0 0) 0x7ffb780b22d0 con 0x7ffb780b2800 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 10 monclient(hunting): my global_id is 4336 2024-03-29T10:57:51.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.688+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb78141eb0 con 0x7ffb780b2800 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613473486 0 0) 0x7ffb78142d30 con 0x7ffb780a9c00 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient(hunting): my global_id is 4397 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb780b22d0 con 0x7ffb780a9c00 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1389472225 0 0) 0x7ffb64003520 con 0x7ffb780b38c0 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 >> v1:172.21.15.22:6790/0 conn(0x7ffb780a9c00 legacy=0x7ffb7813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 >> v1:172.21.15.26:6789/0 conn(0x7ffb780b2800 legacy=0x7ffb78142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: _finish_hunting 0 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 monclient: found mon.noname-a 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb780aaf90 con 0x7ffb780b38c0 2024-03-29T10:57:51.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: _finish_auth 0 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: _check_auth_tickets 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb70001a10 con 0x7ffb780b38c0 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: dump: 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:51.674 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb7e058640 5 monclient: authenticate success, global_id 4344 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb7e058640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb70001b70 con 0x7ffb780b38c0 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 1 -- 172.21.15.22:0/3398978950 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb70001f00 con 0x7ffb780b38c0 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.689+0000 7ffb6ffff640 10 monclient: dump: 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.675 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.690+0000 7ffb7e058640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.690+0000 7ffb7e058640 4 set_mon_vals no callback set 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.690+0000 7ffb7e058640 10 monclient: shutdown 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.690+0000 7ffb7e058640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.690+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 >> v1:172.21.15.22:6789/0 conn(0x7ffb780b38c0 legacy=0x7ffb7814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.690+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 shutdown_connections 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.691+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 >> 172.21.15.22:0/3398978950 conn(0x7ffb78066a00 msgr2=0x7ffb78066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.691+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 shutdown_connections 2024-03-29T10:57:51.676 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.691+0000 7ffb7e058640 1 -- 172.21.15.22:0/3398978950 wait complete. 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.692+0000 7ffb7e058640 10 monclient: build_initial_monmap 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.692+0000 7ffb7e058640 10 monclient: monmap: 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:51.683 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.693+0000 7ffb7e058640 1 librados: starting msgr at 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.693+0000 7ffb7e058640 1 librados: starting objecter 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.693+0000 7ffb7e058640 1 Processor -- start 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.693+0000 7ffb7e058640 1 -- start start 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.693+0000 7ffb7e058640 1 librados: setting wanted keys 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.693+0000 7ffb7e058640 1 librados: calling monclient init 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.693+0000 7ffb7e058640 10 monclient: init 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.694+0000 7ffb7e058640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.694+0000 7ffb7e058640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7e058640 10 monclient(hunting): picked mon.noname-b con 0x7ffb780a9c00 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7e058640 10 monclient(hunting): picked mon.noname-a con 0x7ffb780b2800 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7d056640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7ffb780a9c00 0x7ffb781d74b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:52406/0 (socket says 172.21.15.22:52406) 2024-03-29T10:57:51.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7c855640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ffb780b2800 0x7ffb78137db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:33356/0 (socket says 172.21.15.22:33356) 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7c855640 1 -- 172.21.15.22:0/2346839951 learned_addr learned my addr 172.21.15.22:0/2346839951 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7e058640 10 monclient(hunting): picked mon.noname-c con 0x7ffb780b38c0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7d857640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ffb780b38c0 0x7ffb781e28c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47692/0 (socket says 172.21.15.22:47692) 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb6dffb640 10 client.?.objecter ms_handle_connect 0x7ffb780b2800 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780acf50 con 0x7ffb780b2800 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb780ad1c0 con 0x7ffb780a9c00 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb6dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb6dffb640 10 client.?.objecter ms_handle_connect 0x7ffb780b38c0 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb6dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb7813eff0 con 0x7ffb780b38c0 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995557022 0 0) 0x7ffb780acf50 con 0x7ffb780b2800 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.695+0000 7ffb7e058640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb7e058640 10 monclient(hunting): authenticate will time out at 973.000085s 2024-03-29T10:57:51.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): my global_id is 4347 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb5c004eb0 con 0x7ffb780b2800 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2117172241 0 0) 0x7ffb7813eff0 con 0x7ffb780b38c0 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): _negotiate 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): my global_id is 4400 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb780acf50 con 0x7ffb780b38c0 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 988303296 0 0) 0x7ffb5c004eb0 con 0x7ffb780b2800 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb7813eff0 con 0x7ffb780b2800 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb680019d0 con 0x7ffb780b2800 2024-03-29T10:57:51.686 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): dump: 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 497584874 0 0) 0x7ffb780acf50 con 0x7ffb780b38c0 2024-03-29T10:57:51.687 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb5c004eb0 con 0x7ffb780b38c0 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb70001e80 con 0x7ffb780b38c0 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.696+0000 7ffb6dffb640 10 monclient(hunting): dump: 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:51.688 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 10 client.?.objecter ms_handle_connect 0x7ffb780a9c00 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1896563986 0 0) 0x7ffb5c004eb0 con 0x7ffb780b38c0 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 >> v1:172.21.15.26:6789/0 conn(0x7ffb780a9c00 legacy=0x7ffb781d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 >> v1:172.21.15.22:6789/0 conn(0x7ffb780b2800 legacy=0x7ffb78137db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 10 monclient: _finish_hunting 0 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 1 monclient: found mon.c 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb7813c6e0 con 0x7ffb780b38c0 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 10 monclient: _finish_auth 0 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 10 monclient: _check_auth_tickets 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb7e058640 5 monclient: authenticate success, global_id 4400 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb7e058640 10 monclient: _renew_subs 2024-03-29T10:57:51.689 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb7e058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.697+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb781d4ae0 con 0x7ffb780b38c0 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb7e058640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb7e058640 10 monclient: _renew_subs 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb7e058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb781d7bc0 con 0x7ffb780b38c0 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb7e058640 1 librados: init done 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb70002020 con 0x7ffb780b38c0 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb70004fb0 con 0x7ffb780b38c0 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb6f7fe640 4 set_mon_vals no callback set 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.698+0000 7ffb6dffb640 10 monclient: dump: 2024-03-29T10:57:51.690 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.700+0000 7ffb7e058640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.700+0000 7ffb7e058640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.700+0000 7ffb7e058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.700+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb58005080 con 0x7ffb780b38c0 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.701+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7ffb7001af80 con 0x7ffb780b38c0 2024-03-29T10:57:51.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.701+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7ffb70084de0 con 0x7ffb780b38c0 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.702+0000 7ffb6dffb640 10 client.4400.objecter ms_dispatch 0x7ffb78000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.702+0000 7ffb6dffb640 3 client.4400.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.702+0000 7ffb7d056640 10 monclient: get_auth_request con 0x7ffb5c06dee0 auth_method 0 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.702+0000 7ffb6dffb640 3 client.4400.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.702+0000 7ffb6dffb640 20 client.4400.objecter dump_active .. 0 homeless 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.702+0000 7ffb6dffb640 10 client.4400.objecter ms_handle_connect 0x7ffb5c06dee0 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.708+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb700506c0 con 0x7ffb780b38c0 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.708+0000 7ffb6dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:51.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:51.708+0000 7ffb6dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:52.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.106+0000 7ffb7e058640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:52.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.106+0000 7ffb7e058640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:52.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.106+0000 7ffb7e058640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:52.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.106+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb58005600 con 0x7ffb780b38c0 2024-03-29T10:57:52.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.107+0000 7ffb6dffb640 1 -- 172.21.15.22:0/2346839951 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7ffb700525f0 con 0x7ffb780b38c0 2024-03-29T10:57:52.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.107+0000 7ffb6dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:52.088 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.107+0000 7ffb6dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:52.088 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:52.088 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:52.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 10 librados: watch_flush enter 2024-03-29T10:57:52.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 10 librados: watch_flush exit 2024-03-29T10:57:52.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 20 client.4400.objecter shutdown clearing up homeless session... 2024-03-29T10:57:52.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 10 client.4400.objecter successfully canceled tick 2024-03-29T10:57:52.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 >> v1:172.21.15.22:6816/380221644 conn(0x7ffb5c06dee0 legacy=0x7ffb5c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:52.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 10 monclient: shutdown 2024-03-29T10:57:52.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:52.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 >> v1:172.21.15.22:6790/0 conn(0x7ffb780b38c0 legacy=0x7ffb781e28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:52.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 shutdown_connections 2024-03-29T10:57:52.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.111+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 >> 172.21.15.22:0/2346839951 conn(0x7ffb78066a00 msgr2=0x7ffb78140520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:52.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.112+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 shutdown_connections 2024-03-29T10:57:52.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.112+0000 7ffb7e058640 1 -- 172.21.15.22:0/2346839951 wait complete. 2024-03-29T10:57:52.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:52.112+0000 7ffb7e058640 1 librados: shutdown 2024-03-29T10:57:52.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:57:58.914 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.087+0000 7fad8a2c9640 10 monclient: get_monmap_and_config 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.087+0000 7fad8a2c9640 10 monclient: build_initial_monmap 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.087+0000 7fad8a2c9640 10 monclient: monmap: 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:57:59.088176+0000 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:57:59.088176+0000 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:59.068 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:57:59.069 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:57:59.069 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:59.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.089+0000 7fad8a2c9640 1 Processor -- start 2024-03-29T10:57:59.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.089+0000 7fad8a2c9640 1 -- start start 2024-03-29T10:57:59.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.089+0000 7fad8a2c9640 10 monclient: init 2024-03-29T10:57:59.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.089+0000 7fad8a2c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:59.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.089+0000 7fad8a2c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:57:59.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 10 monclient(hunting): picked mon.noname-c con 0x7fad7c038c60 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 10 monclient(hunting): picked mon.noname-b con 0x7fad7c0324b0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fad7c0d11e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c0caf80 con 0x7fad7c0d11e0 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c039850 con 0x7fad7c038c60 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c0c78d0 con 0x7fad7c0324b0 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad8a2c9640 10 monclient(hunting): authenticate will time out at 980.394190s 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad837fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fad7c0324b0 0x7fad7c0cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49738/0 (socket says 172.21.15.22:49738) 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad837fe640 1 -- 172.21.15.22:0/3489571637 learned_addr learned my addr 172.21.15.22:0/3489571637 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:59.072 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790223454 0 0) 0x7fad7c0c78d0 con 0x7fad7c0324b0 2024-03-29T10:57:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad82ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:57:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad82ffd640 10 monclient(hunting): my global_id is 4403 2024-03-29T10:57:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.090+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad74003520 con 0x7fad7c0324b0 2024-03-29T10:57:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.092+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3397062747 0 0) 0x7fad74003520 con 0x7fad7c0324b0 2024-03-29T10:57:59.073 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.092+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 >> v1:172.21.15.26:6789/0 conn(0x7fad7c038c60 legacy=0x7fad7c0c4ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:57:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 >> v1:172.21.15.22:6789/0 conn(0x7fad7c0d11e0 legacy=0x7fad7c0d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:57:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 1 monclient: found mon.noname-b 2024-03-29T10:57:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:59.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c0c7a60 con 0x7fad7c0324b0 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 10 monclient: _finish_auth 0 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fad70001690 con 0x7fad7c0324b0 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.094+0000 7fad82ffd640 10 monclient: dump: 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:59.076 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:59.077 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:59.077 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:59.077 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:59.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.095+0000 7fad8a2c9640 5 monclient: authenticate success, global_id 4403 2024-03-29T10:57:59.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.095+0000 7fad8a2c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:57:59.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad700018f0 con 0x7fad7c0324b0 2024-03-29T10:57:59.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:59.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad8a2c9640 10 monclient: get_monmap_and_config success 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad8a2c9640 4 set_mon_vals no callback set 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad82ffd640 1 -- 172.21.15.22:0/3489571637 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fad70001d00 con 0x7fad7c0324b0 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad82ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad82ffd640 10 monclient: dump: 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:59.079 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad8a2c9640 10 monclient: shutdown 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad8a2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.096+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3489571637 >> v1:172.21.15.22:6790/0 conn(0x7fad7c0324b0 legacy=0x7fad7c0cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.097+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3489571637 shutdown_connections 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.097+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3489571637 >> 172.21.15.22:0/3489571637 conn(0x7fad7c005c20 msgr2=0x7fad7c006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.097+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3489571637 shutdown_connections 2024-03-29T10:57:59.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.097+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3489571637 wait complete. 2024-03-29T10:57:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.098+0000 7fad8a2c9640 10 monclient: build_initial_monmap 2024-03-29T10:57:59.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.098+0000 7fad8a2c9640 10 monclient: monmap: 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:57:59.084 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.098+0000 7fad8a2c9640 1 librados: starting msgr at 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.098+0000 7fad8a2c9640 1 librados: starting objecter 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.098+0000 7fad8a2c9640 1 Processor -- start 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.099+0000 7fad8a2c9640 1 -- start start 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.099+0000 7fad8a2c9640 1 librados: setting wanted keys 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.099+0000 7fad8a2c9640 1 librados: calling monclient init 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.099+0000 7fad8a2c9640 10 monclient: init 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 10 monclient(hunting): picked mon.noname-c con 0x7fad7c0324b0 addr v1:172.21.15.22:6790/0 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 10 monclient(hunting): picked mon.noname-a con 0x7fad7c038c60 addr v1:172.21.15.22:6789/0 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 10 monclient(hunting): picked mon.noname-b con 0x7fad7c0d11e0 addr v1:172.21.15.26:6789/0 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c036370 con 0x7fad7c038c60 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c167380 con 0x7fad7c0d11e0 2024-03-29T10:57:59.085 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad7c037190 con 0x7fad7c0324b0 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.100+0000 7fad8a2c9640 10 monclient(hunting): authenticate will time out at 980.404866s 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.101+0000 7fad83fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fad7c0324b0 0x7fad7c0c2810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:49952/0 (socket says 172.21.15.22:49952) 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.101+0000 7fad83fff640 1 -- 172.21.15.22:0/3836213237 learned_addr learned my addr 172.21.15.22:0/3836213237 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 10 client.?.objecter ms_handle_connect 0x7fad7c0324b0 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434856135 0 0) 0x7fad7c037190 con 0x7fad7c0324b0 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 10 monclient(hunting): my global_id is 4406 2024-03-29T10:57:59.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.103+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad68004eb0 con 0x7fad7c0324b0 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.104+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3269008418 0 0) 0x7fad68004eb0 con 0x7fad7c0324b0 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.104+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad7c037190 con 0x7fad7c0324b0 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.104+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fad78001870 con 0x7fad7c0324b0 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.104+0000 7fad80ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.104+0000 7fad80ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.104+0000 7fad80ff9640 10 monclient(hunting): dump: 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.090 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.104+0000 7fad80ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 client.?.objecter ms_handle_connect 0x7fad7c0d11e0 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2945966764 0 0) 0x7fad7c037190 con 0x7fad7c0324b0 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 >> v1:172.21.15.26:6789/0 conn(0x7fad7c0d11e0 legacy=0x7fad7c166c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 >> v1:172.21.15.22:6789/0 conn(0x7fad7c038c60 legacy=0x7fad7c166560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 1 monclient: found mon.c 2024-03-29T10:57:59.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad7c0343d0 con 0x7fad7c0324b0 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: _finish_auth 0 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 client.?.objecter ms_handle_connect 0x7fad7c038c60 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad78004900 con 0x7fad7c0324b0 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fad78001870 con 0x7fad7c0324b0 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.105+0000 7fad80ff9640 10 monclient: dump: 2024-03-29T10:57:59.092 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 5 monclient: authenticate success, global_id 4406 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 10 monclient: _renew_subs 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad7c0347b0 con 0x7fad7c0324b0 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:57:59.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 10 monclient: _renew_subs 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad7c0c5070 con 0x7fad7c0324b0 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.106+0000 7fad8a2c9640 1 librados: init done 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.107+0000 7fad827fc640 4 set_mon_vals no callback set 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.108+0000 7fad8a2c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.108+0000 7fad8a2c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.108+0000 7fad8a2c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.108+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad64005080 con 0x7fad7c0324b0 2024-03-29T10:57:59.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.109+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fad7801a8e0 con 0x7fad7c0324b0 2024-03-29T10:57:59.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.110+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fad78054630 con 0x7fad7c0324b0 2024-03-29T10:57:59.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.110+0000 7fad80ff9640 10 client.4406.objecter ms_dispatch 0x7fad84000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:57:59.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.110+0000 7fad80ff9640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:57:59.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.110+0000 7fad80ff9640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:57:59.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.110+0000 7fad80ff9640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-29T10:57:59.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.110+0000 7fad837fe640 10 monclient: get_auth_request con 0x7fad6806dd60 auth_method 0 2024-03-29T10:57:59.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.110+0000 7fad80ff9640 10 client.4406.objecter ms_handle_connect 0x7fad6806dd60 2024-03-29T10:57:59.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.120+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad780508d0 con 0x7fad7c0324b0 2024-03-29T10:57:59.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.120+0000 7fad80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:57:59.100 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.120+0000 7fad80ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:57:59.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.454+0000 7fad8a2c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:59.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.454+0000 7fad8a2c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:59.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.454+0000 7fad8a2c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:57:59.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.454+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad64005330 con 0x7fad7c0324b0 2024-03-29T10:57:59.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.455+0000 7fad80ff9640 1 -- 172.21.15.22:0/3836213237 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fad780508d0 con 0x7fad7c0324b0 2024-03-29T10:57:59.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.455+0000 7fad80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:57:59.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.455+0000 7fad80ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:57:59.436 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:57:59.436 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:57:59.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 10 librados: watch_flush enter 2024-03-29T10:57:59.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 10 librados: watch_flush exit 2024-03-29T10:57:59.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-29T10:57:59.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 10 client.4406.objecter successfully canceled tick 2024-03-29T10:57:59.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 >> v1:172.21.15.22:6816/380221644 conn(0x7fad6806dd60 legacy=0x7fad68070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:59.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 10 monclient: shutdown 2024-03-29T10:57:59.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:57:59.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 >> v1:172.21.15.22:6790/0 conn(0x7fad7c0324b0 legacy=0x7fad7c0c2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:57:59.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 shutdown_connections 2024-03-29T10:57:59.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.459+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 >> 172.21.15.22:0/3836213237 conn(0x7fad7c005c20 msgr2=0x7fad7c03ae60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:57:59.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.460+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 shutdown_connections 2024-03-29T10:57:59.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.460+0000 7fad8a2c9640 1 -- 172.21.15.22:0/3836213237 wait complete. 2024-03-29T10:57:59.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:57:59.460+0000 7fad8a2c9640 1 librados: shutdown 2024-03-29T10:57:59.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:06.263 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:06.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.435+0000 7f620e489640 10 monclient: get_monmap_and_config 2024-03-29T10:58:06.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.435+0000 7f620e489640 10 monclient: build_initial_monmap 2024-03-29T10:58:06.415 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.435+0000 7f620e489640 10 monclient: monmap: 2024-03-29T10:58:06.415 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:06.415 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:06.415 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:06.435839+0000 2024-03-29T10:58:06.416 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:06.435839+0000 2024-03-29T10:58:06.416 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:06.416 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:06.416 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:06.416 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:06.416 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:06.416 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:06.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.436+0000 7f620e489640 1 Processor -- start 2024-03-29T10:58:06.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.436+0000 7f620e489640 1 -- start start 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.436+0000 7f620e489640 10 monclient: init 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.437+0000 7f620e489640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.437+0000 7f620e489640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.437+0000 7f620e489640 10 monclient(hunting): picked mon.noname-b con 0x7f62000389e0 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.437+0000 7f620d487640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f62000389e0 0x7f6200033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37420/0 (socket says 172.21.15.22:37420) 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.437+0000 7f620d487640 1 -- 172.21.15.22:0/2825333115 learned_addr learned my addr 172.21.15.22:0/2825333115 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.437+0000 7f620e489640 10 monclient(hunting): picked mon.noname-c con 0x7f6200038dc0 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.438+0000 7f620e489640 10 monclient(hunting): picked mon.noname-a con 0x7f6200030660 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.438+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000c54c0 con 0x7f6200030660 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.438+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6200036430 con 0x7f6200038dc0 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.438+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6200039840 con 0x7f62000389e0 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.438+0000 7f620e489640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:06.419 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.438+0000 7f620e489640 10 monclient(hunting): authenticate will time out at 987.742418s 2024-03-29T10:58:06.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.441+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3537773827 0 0) 0x7f6200039840 con 0x7f62000389e0 2024-03-29T10:58:06.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.441+0000 7f61fffff640 10 monclient(hunting): _negotiate 2024-03-29T10:58:06.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.441+0000 7f61fffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:06.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.441+0000 7f61fffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:06.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.441+0000 7f61fffff640 10 monclient(hunting): my global_id is 4409 2024-03-29T10:58:06.421 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.441+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61f4003520 con 0x7f62000389e0 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.444+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834139525 0 0) 0x7f61f4003520 con 0x7f62000389e0 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.444+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 >> v1:172.21.15.26:6789/0 conn(0x7f6200038dc0 legacy=0x7f62000cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.444+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 >> v1:172.21.15.22:6789/0 conn(0x7f6200030660 legacy=0x7f62000d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: _finish_hunting 0 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 1 monclient: found mon.noname-b 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:58:06.425 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6200035690 con 0x7f62000389e0 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: _finish_auth 0 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: _check_auth_tickets 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6204001690 con 0x7f62000389e0 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: dump: 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:06.426 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f620e489640 5 monclient: authenticate success, global_id 4409 2024-03-29T10:58:06.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f620e489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:06.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6204001860 con 0x7f62000389e0 2024-03-29T10:58:06.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:06.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 1 -- 172.21.15.22:0/2825333115 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6204001c70 con 0x7f62000389e0 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f61fffff640 10 monclient: dump: 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:06.429 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f620e489640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.445+0000 7f620e489640 4 set_mon_vals no callback set 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.446+0000 7f620e489640 10 monclient: shutdown 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.446+0000 7f620e489640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.446+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 >> v1:172.21.15.22:6790/0 conn(0x7f62000389e0 legacy=0x7f6200033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:06.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.446+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 shutdown_connections 2024-03-29T10:58:06.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.446+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 >> 172.21.15.22:0/2825333115 conn(0x7f6200005c00 msgr2=0x7f6200006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:06.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.447+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 shutdown_connections 2024-03-29T10:58:06.431 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.448+0000 7f620e489640 1 -- 172.21.15.22:0/2825333115 wait complete. 2024-03-29T10:58:06.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.449+0000 7f620e489640 10 monclient: build_initial_monmap 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.449+0000 7f620e489640 10 monclient: monmap: 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.450+0000 7f620e489640 1 librados: starting msgr at 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.450+0000 7f620e489640 1 librados: starting objecter 2024-03-29T10:58:06.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.450+0000 7f620e489640 1 Processor -- start 2024-03-29T10:58:06.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.450+0000 7f620e489640 1 -- start start 2024-03-29T10:58:06.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.450+0000 7f620e489640 1 librados: setting wanted keys 2024-03-29T10:58:06.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.451+0000 7f620e489640 1 librados: calling monclient init 2024-03-29T10:58:06.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.451+0000 7f620e489640 10 monclient: init 2024-03-29T10:58:06.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.452+0000 7f620e489640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:06.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.452+0000 7f620e489640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:06.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.452+0000 7f620e489640 10 monclient(hunting): picked mon.noname-a con 0x7f6200030660 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620e489640 10 monclient(hunting): picked mon.noname-b con 0x7f62000389e0 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620e489640 10 monclient(hunting): picked mon.noname-c con 0x7f62000c8480 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620e489640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620002d150 con 0x7f6200030660 2024-03-29T10:58:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620e489640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620002f3e0 con 0x7f62000389e0 2024-03-29T10:58:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620e489640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62000c0400 con 0x7f62000c8480 2024-03-29T10:58:06.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620e489640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620e489640 10 monclient(hunting): authenticate will time out at 987.758229s 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620dc88640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f62000c8480 0x7f6200167a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37750/0 (socket says 172.21.15.22:37750) 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f620dc88640 1 -- 172.21.15.22:0/4271188823 learned_addr learned my addr 172.21.15.22:0/4271188823 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f61fdffb640 10 client.?.objecter ms_handle_connect 0x7f62000c8480 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.454+0000 7f61fdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192836736 0 0) 0x7f62000c0400 con 0x7f62000c8480 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): _negotiate 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): my global_id is 4412 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61ec004eb0 con 0x7f62000c8480 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1215803934 0 0) 0x7f61ec004eb0 con 0x7f62000c8480 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f62000c0400 con 0x7f62000c8480 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f620804fdd0 con 0x7f62000c8480 2024-03-29T10:58:06.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): dump: 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:06.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 177451996 0 0) 0x7f62000c0400 con 0x7f62000c8480 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 >> v1:172.21.15.26:6789/0 conn(0x7f62000389e0 legacy=0x7f620015d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.455+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 >> v1:172.21.15.22:6789/0 conn(0x7f6200030660 legacy=0x7f620002fb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 10 monclient: _finish_hunting 0 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 1 monclient: found mon.c 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620015da60 con 0x7f62000c8480 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 10 monclient: _finish_auth 0 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 10 monclient: _check_auth_tickets 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f620804f400 con 0x7f62000c8480 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f620e489640 5 monclient: authenticate success, global_id 4412 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f620e489640 10 monclient: _renew_subs 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f620e489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:06.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.457+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620015dd10 con 0x7f62000c8480 2024-03-29T10:58:06.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f620e489640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:06.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f61fdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f61ff7fe640 4 set_mon_vals no callback set 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6208075710 con 0x7f62000c8480 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f61fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f61fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f61fdffb640 10 monclient: dump: 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:06.443 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f620e489640 10 monclient: _renew_subs 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f620e489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6200169360 con 0x7f62000c8480 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.458+0000 7f620e489640 1 librados: init done 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.459+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f6208067bf0 con 0x7f62000c8480 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f620d487640 10 monclient: get_auth_request con 0x7f61ec06dad0 auth_method 0 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f62080ff5d0 con 0x7f62000c8480 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f61fdffb640 10 client.4412.objecter ms_dispatch 0x7f6208000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f61fdffb640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:06.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f61fdffb640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f61fdffb640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-29T10:58:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f620e489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f620e489640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.460+0000 7f620e489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.461+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620015d070 con 0x7f62000c8480 2024-03-29T10:58:06.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.461+0000 7f61fdffb640 10 client.4412.objecter ms_handle_connect 0x7f61ec06dad0 2024-03-29T10:58:06.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.468+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f620015d070 con 0x7f62000c8480 2024-03-29T10:58:06.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.468+0000 7f61fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:06.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.468+0000 7f61fdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:06.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.795+0000 7f620e489640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:06.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.795+0000 7f620e489640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:06.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.795+0000 7f620e489640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:06.775 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.795+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f620015d070 con 0x7f62000c8480 2024-03-29T10:58:06.776 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.796+0000 7f61fdffb640 1 -- 172.21.15.22:0/4271188823 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f620015d070 con 0x7f62000c8480 2024-03-29T10:58:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.796+0000 7f61fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:06.777 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.796+0000 7f61fdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:06.777 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:06.777 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.800+0000 7f620e489640 10 librados: watch_flush enter 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.800+0000 7f620e489640 10 librados: watch_flush exit 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.800+0000 7f620e489640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.800+0000 7f620e489640 10 client.4412.objecter successfully canceled tick 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.800+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 >> v1:172.21.15.22:6816/380221644 conn(0x7f61ec06dad0 legacy=0x7f61ec06ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.800+0000 7f620e489640 10 monclient: shutdown 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.800+0000 7f620e489640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:06.781 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.801+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 >> v1:172.21.15.22:6790/0 conn(0x7f62000c8480 legacy=0x7f6200167a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:06.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.801+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 shutdown_connections 2024-03-29T10:58:06.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.801+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 >> 172.21.15.22:0/4271188823 conn(0x7f6200005c00 msgr2=0x7f6200034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:06.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.801+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 shutdown_connections 2024-03-29T10:58:06.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.801+0000 7f620e489640 1 -- 172.21.15.22:0/4271188823 wait complete. 2024-03-29T10:58:06.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:06.802+0000 7f620e489640 1 librados: shutdown 2024-03-29T10:58:06.800 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:13.606 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.779+0000 7f53cc4c9640 10 monclient: get_monmap_and_config 2024-03-29T10:58:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.779+0000 7f53cc4c9640 10 monclient: build_initial_monmap 2024-03-29T10:58:13.763 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.779+0000 7f53cc4c9640 10 monclient: monmap: 2024-03-29T10:58:13.763 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:13.763 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:13.763 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:13.780303+0000 2024-03-29T10:58:13.763 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:13.780303+0000 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.780+0000 7f53cc4c9640 1 Processor -- start 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.780+0000 7f53cc4c9640 1 -- start start 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.780+0000 7f53cc4c9640 10 monclient: init 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.781+0000 7f53cc4c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f53c41422b0 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f53c4142690 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f53c40b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:13.764 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c4142c80 con 0x7f53c4142690 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c40aae40 con 0x7f53c41422b0 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c413fbd0 con 0x7f53c40b17d0 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53cc4c9640 10 monclient(hunting): authenticate will time out at 995.086838s 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53caa66640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f53c40b17d0 0x7f53c40b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:51094/0 (socket says 172.21.15.22:51094) 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.782+0000 7f53caa66640 1 -- 172.21.15.22:0/872150779 learned_addr learned my addr 172.21.15.22:0/872150779 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.783+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3669248874 0 0) 0x7f53c413fbd0 con 0x7f53c40b17d0 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.783+0000 7f53c9263640 10 monclient(hunting): _negotiate 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.783+0000 7f53c9263640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.783+0000 7f53c9263640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.783+0000 7f53c9263640 10 monclient(hunting): my global_id is 4415 2024-03-29T10:58:13.765 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.783+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53b0003520 con 0x7f53c40b17d0 2024-03-29T10:58:13.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.784+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4032345612 0 0) 0x7f53b0003520 con 0x7f53c40b17d0 2024-03-29T10:58:13.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.784+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 >> v1:172.21.15.26:6789/0 conn(0x7f53c41422b0 legacy=0x7f53c413f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:13.766 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.786+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 >> v1:172.21.15.22:6789/0 conn(0x7f53c4142690 legacy=0x7f53c40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 10 monclient: _finish_hunting 0 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 1 monclient: found mon.noname-b 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c4151940 con 0x7f53c40b17d0 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 10 monclient: _finish_auth 0 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 10 monclient: _check_auth_tickets 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53bc001a10 con 0x7f53c40b17d0 2024-03-29T10:58:13.767 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53c9263640 10 monclient: dump: 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53cc4c9640 5 monclient: authenticate success, global_id 4415 2024-03-29T10:58:13.768 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.787+0000 7f53cc4c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53bc001b70 con 0x7f53c40b17d0 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53c9263640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53c9263640 1 -- 172.21.15.22:0/872150779 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53bc001fc0 con 0x7f53c40b17d0 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53c9263640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53c9263640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53c9263640 10 monclient: dump: 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.769 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.770 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:13.770 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:13.770 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:13.770 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:13.770 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53cc4c9640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:13.770 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.788+0000 7f53cc4c9640 4 set_mon_vals no callback set 2024-03-29T10:58:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.789+0000 7f53cc4c9640 10 monclient: shutdown 2024-03-29T10:58:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.789+0000 7f53cc4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:13.771 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.789+0000 7f53cc4c9640 1 -- 172.21.15.22:0/872150779 >> v1:172.21.15.22:6790/0 conn(0x7f53c40b17d0 legacy=0x7f53c40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.789+0000 7f53cc4c9640 1 -- 172.21.15.22:0/872150779 shutdown_connections 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.789+0000 7f53cc4c9640 1 -- 172.21.15.22:0/872150779 >> 172.21.15.22:0/872150779 conn(0x7f53c4066a00 msgr2=0x7f53c4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.789+0000 7f53cc4c9640 1 -- 172.21.15.22:0/872150779 shutdown_connections 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.789+0000 7f53cc4c9640 1 -- 172.21.15.22:0/872150779 wait complete. 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.790+0000 7f53cc4c9640 10 monclient: build_initial_monmap 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.790+0000 7f53cc4c9640 10 monclient: monmap: 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:13.772 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.791+0000 7f53cc4c9640 1 librados: starting msgr at 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.791+0000 7f53cc4c9640 1 librados: starting objecter 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.791+0000 7f53cc4c9640 1 Processor -- start 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 1 -- start start 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 1 librados: setting wanted keys 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 1 librados: calling monclient init 2024-03-29T10:58:13.773 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 10 monclient: init 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 10 monclient(hunting): picked mon.noname-a con 0x7f53c40b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 10 monclient(hunting): picked mon.noname-c con 0x7f53c41422b0 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.793+0000 7f53cc4c9640 10 monclient(hunting): picked mon.noname-b con 0x7f53c4142690 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.794+0000 7f53cc4c9640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c412a7c0 con 0x7f53c40b17d0 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.794+0000 7f53cc4c9640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c40acfd0 con 0x7f53c4142690 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.794+0000 7f53cc4c9640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c40ab790 con 0x7f53c41422b0 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.794+0000 7f53cc4c9640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:13.782 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.794+0000 7f53cc4c9640 10 monclient(hunting): authenticate will time out at 995.098116s 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53c9a64640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f53c41422b0 0x7f53c41299a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:51312/0 (socket says 172.21.15.22:51312) 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53c9a64640 1 -- 172.21.15.22:0/1172687213 learned_addr learned my addr 172.21.15.22:0/1172687213 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 client.?.objecter ms_handle_connect 0x7f53c41422b0 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 client.?.objecter ms_handle_connect 0x7f53c40b17d0 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676903253 0 0) 0x7f53c40ab790 con 0x7f53c41422b0 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 10 monclient(hunting): my global_id is 4418 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.797+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a8004eb0 con 0x7f53c41422b0 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272038389 0 0) 0x7f53c412a7c0 con 0x7f53c40b17d0 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:13.783 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): my global_id is 4356 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c40ab790 con 0x7f53c40b17d0 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3066772146 0 0) 0x7f53a8004eb0 con 0x7f53c41422b0 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53c412a7c0 con 0x7f53c41422b0 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53b4001930 con 0x7f53c41422b0 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): dump: 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.784 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2753360361 0 0) 0x7f53c40ab790 con 0x7f53c40b17d0 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53a8004eb0 con 0x7f53c40b17d0 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53c00017b0 con 0x7f53c40b17d0 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.798+0000 7f53baffd640 10 monclient(hunting): dump: 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.785 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2504284627 0 0) 0x7f53c412a7c0 con 0x7f53c41422b0 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 >> v1:172.21.15.26:6789/0 conn(0x7f53c4142690 legacy=0x7f53c412a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 >> v1:172.21.15.22:6789/0 conn(0x7f53c40b17d0 legacy=0x7f53c41da890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 1 monclient: found mon.c 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:13.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c41dafa0 con 0x7f53c41422b0 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 5 monclient: authenticate success, global_id 4418 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 10 monclient: _renew_subs 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c41db190 con 0x7f53c41422b0 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 10 monclient: _renew_subs 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c40b79a0 con 0x7f53c41422b0 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.799+0000 7f53cc4c9640 1 librados: init done 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.800+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53b4001a90 con 0x7f53c41422b0 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.800+0000 7f53baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:13.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.800+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53b4003870 con 0x7f53c41422b0 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.800+0000 7f53baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.800+0000 7f53baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.800+0000 7f53baffd640 10 monclient: dump: 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:13.788 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:13.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.802+0000 7f53c8a62640 4 set_mon_vals no callback set 2024-03-29T10:58:13.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.802+0000 7f53cc4c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:13.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.803+0000 7f53cc4c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:13.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.803+0000 7f53cc4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.803+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53ac005080 con 0x7f53c41422b0 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.805+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f53b40039d0 con 0x7f53c41422b0 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.806+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f53b40849b0 con 0x7f53c41422b0 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.806+0000 7f53baffd640 10 client.4418.objecter ms_dispatch 0x7f53c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.806+0000 7f53baffd640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.806+0000 7f53baffd640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.806+0000 7f53baffd640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T10:58:13.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.810+0000 7f53ca265640 10 monclient: get_auth_request con 0x7f53a806ddf0 auth_method 0 2024-03-29T10:58:13.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.811+0000 7f53baffd640 10 client.4418.objecter ms_handle_connect 0x7f53a806ddf0 2024-03-29T10:58:13.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.815+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53b40500c0 con 0x7f53c41422b0 2024-03-29T10:58:13.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.815+0000 7f53baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:13.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:13.815+0000 7f53baffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:14.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.139+0000 7f53cc4c9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:14.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.139+0000 7f53cc4c9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:14.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.139+0000 7f53cc4c9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:14.120 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.139+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53ac005330 con 0x7f53c41422b0 2024-03-29T10:58:14.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.142+0000 7f53baffd640 1 -- 172.21.15.22:0/1172687213 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f53b40521c0 con 0x7f53c41422b0 2024-03-29T10:58:14.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.142+0000 7f53baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:14.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.142+0000 7f53baffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:14.124 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:14.124 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:14.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.146+0000 7f53cc4c9640 10 librados: watch_flush enter 2024-03-29T10:58:14.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.146+0000 7f53cc4c9640 10 librados: watch_flush exit 2024-03-29T10:58:14.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.146+0000 7f53cc4c9640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T10:58:14.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.146+0000 7f53cc4c9640 10 client.4418.objecter successfully canceled tick 2024-03-29T10:58:14.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.146+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 >> v1:172.21.15.22:6816/380221644 conn(0x7f53a806ddf0 legacy=0x7f53a80702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:14.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.146+0000 7f53cc4c9640 10 monclient: shutdown 2024-03-29T10:58:14.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.147+0000 7f53cc4c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:14.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.147+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 >> v1:172.21.15.22:6790/0 conn(0x7f53c41422b0 legacy=0x7f53c41299a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:14.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.147+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 shutdown_connections 2024-03-29T10:58:14.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.147+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 >> 172.21.15.22:0/1172687213 conn(0x7f53c4066a00 msgr2=0x7f53c40b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:14.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.147+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 shutdown_connections 2024-03-29T10:58:14.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.147+0000 7f53cc4c9640 1 -- 172.21.15.22:0/1172687213 wait complete. 2024-03-29T10:58:14.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:14.147+0000 7f53cc4c9640 1 librados: shutdown 2024-03-29T10:58:14.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:20.954 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:21.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.123+0000 7f2ea080d640 10 monclient: get_monmap_and_config 2024-03-29T10:58:21.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.123+0000 7f2ea080d640 10 monclient: build_initial_monmap 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.123+0000 7f2ea080d640 10 monclient: monmap: 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:21.124051+0000 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:21.124051+0000 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.124+0000 7f2ea080d640 1 Processor -- start 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.124+0000 7f2ea080d640 1 -- start start 2024-03-29T10:58:21.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.124+0000 7f2ea080d640 10 monclient: init 2024-03-29T10:58:21.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.125+0000 7f2ea080d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.125+0000 7f2ea080d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.125+0000 7f2ea080d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e9c06d9d0 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2e99d9b640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f2e9c13e820 0x7f2e9c13ec50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:45721/0 (socket says 172.21.15.22:45721) 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2e99d9b640 1 -- 172.21.15.22:0/3381061443 learned_addr learned my addr 172.21.15.22:0/3381061443 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2ea080d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e9c13e820 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2ea080d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9c0af1d0 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c0aed10 con 0x7f2e9c13e820 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c0afd50 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c0ab490 con 0x7f2e9c0af1d0 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2ea080d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.128+0000 7f2ea080d640 10 monclient(hunting): authenticate will time out at 1002.433036s 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.129+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104677545 0 0) 0x7f2e9c0aed10 con 0x7f2e9c13e820 2024-03-29T10:58:21.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.130+0000 7f2e9959a640 10 monclient(hunting): _negotiate 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.130+0000 7f2e9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.130+0000 7f2e9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.130+0000 7f2e9959a640 10 monclient(hunting): my global_id is 4359 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.130+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e84003520 con 0x7f2e9c13e820 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257220753 0 0) 0x7f2e9c0afd50 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient(hunting): _negotiate 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient(hunting): my global_id is 4348 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e9c0aed10 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 673028460 0 0) 0x7f2e84003520 con 0x7f2e9c13e820 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 >> v1:172.21.15.22:6790/0 conn(0x7f2e9c0af1d0 legacy=0x7f2e9c0af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 >> v1:172.21.15.26:6789/0 conn(0x7f2e9c06d9d0 legacy=0x7f2e9c0b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient: _finish_hunting 0 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 monclient: found mon.noname-a 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c0ab770 con 0x7f2e9c13e820 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient: _finish_auth 0 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient: _check_auth_tickets 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2e90001690 con 0x7f2e9c13e820 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2e9959a640 10 monclient: dump: 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:21.147 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2ea080d640 5 monclient: authenticate success, global_id 4359 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.131+0000 7f2ea080d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e900017f0 con 0x7f2e9c13e820 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2e9959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2e9959a640 1 -- 172.21.15.22:0/3381061443 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2e90001c40 con 0x7f2e9c13e820 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2e9959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:21.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2e9959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2e9959a640 10 monclient: dump: 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2ea080d640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.132+0000 7f2ea080d640 4 set_mon_vals no callback set 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.133+0000 7f2ea080d640 10 monclient: shutdown 2024-03-29T10:58:21.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.133+0000 7f2ea080d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.133+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 >> v1:172.21.15.22:6789/0 conn(0x7f2e9c13e820 legacy=0x7f2e9c13ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.133+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 shutdown_connections 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.133+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 >> 172.21.15.22:0/3381061443 conn(0x7f2e9c06b660 msgr2=0x7f2e9c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.134+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 shutdown_connections 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.134+0000 7f2ea080d640 1 -- 172.21.15.22:0/3381061443 wait complete. 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.136+0000 7f2ea080d640 10 monclient: build_initial_monmap 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.136+0000 7f2ea080d640 10 monclient: monmap: 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:21.150 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.136+0000 7f2ea080d640 1 librados: starting msgr at 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.136+0000 7f2ea080d640 1 librados: starting objecter 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.137+0000 7f2ea080d640 1 Processor -- start 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.137+0000 7f2ea080d640 1 -- start start 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.137+0000 7f2ea080d640 1 librados: setting wanted keys 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.137+0000 7f2ea080d640 1 librados: calling monclient init 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.137+0000 7f2ea080d640 10 monclient: init 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e9c06d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e9c0af1d0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:21.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e9c13e820 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c1d9740 con 0x7f2e9c0af1d0 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c137b60 con 0x7f2e9c13e820 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e9c06ec40 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.138+0000 7f2ea080d640 10 monclient(hunting): authenticate will time out at 1002.442823s 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e9a59c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2e9c06d9d0 0x7f2e9c1382b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58194/0 (socket says 172.21.15.22:58194) 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e9a59c640 1 -- 172.21.15.22:0/1047989905 learned_addr learned my addr 172.21.15.22:0/1047989905 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 10 client.?.objecter ms_handle_connect 0x7f2e9c06d9d0 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445684957 0 0) 0x7f2e9c06ec40 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 10 monclient(hunting): my global_id is 4424 2024-03-29T10:58:21.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.139+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e7c004eb0 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.140+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3864535790 0 0) 0x7f2e7c004eb0 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.140+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e9c06ec40 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.140+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2e8c001a20 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.140+0000 7f2e8affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.140+0000 7f2e8affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.140+0000 7f2e8affd640 10 monclient(hunting): dump: 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:21.153 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.140+0000 7f2e8affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.142+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2966002348 0 0) 0x7f2e9c06ec40 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.142+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 >> v1:172.21.15.26:6789/0 conn(0x7f2e9c13e820 legacy=0x7f2e9c1d9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 >> v1:172.21.15.22:6789/0 conn(0x7f2e9c0af1d0 legacy=0x7f2e9c1d8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 1 monclient: found mon.c 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e9c06ef20 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 10 monclient: _finish_auth 0 2024-03-29T10:58:21.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 10 client.?.objecter ms_handle_connect 0x7f2e9c13e820 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2e8affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 5 monclient: authenticate success, global_id 4424 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 10 monclient: _renew_subs 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e9c06ddb0 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 10 monclient: _renew_subs 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e9c135350 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.143+0000 7f2ea080d640 1 librados: init done 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.145+0000 7f2ea080d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.145+0000 7f2ea080d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:21.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.145+0000 7f2ea080d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.145+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e9c137940 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.147+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e8c001d00 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.147+0000 7f2e8affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.147+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2e8c003a60 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.147+0000 7f2e8affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.147+0000 7f2e8affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.147+0000 7f2e8affd640 10 monclient: dump: 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:21.156 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.148+0000 7f2e98d99640 4 set_mon_vals no callback set 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.151+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f2e8c003bc0 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.152+0000 7f2e99d9b640 10 monclient: get_auth_request con 0x7f2e7c06dbd0 auth_method 0 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.152+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f2e8c084b00 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.152+0000 7f2e8affd640 10 client.4424.objecter ms_dispatch 0x7f2e9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.152+0000 7f2e8affd640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.152+0000 7f2e8affd640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.152+0000 7f2e8affd640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.153+0000 7f2e8affd640 10 client.4424.objecter ms_handle_connect 0x7f2e7c06dbd0 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.154+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e8c050210 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.154+0000 7f2e8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:21.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.154+0000 7f2e8affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:21.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.501+0000 7f2ea080d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:21.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.501+0000 7f2ea080d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:21.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.501+0000 7f2ea080d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:21.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.501+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e9c075f10 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.482 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:21.483 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:21.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.502+0000 7f2e8affd640 1 -- 172.21.15.22:0/1047989905 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f2e8c052310 con 0x7f2e9c06d9d0 2024-03-29T10:58:21.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.502+0000 7f2e8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:21.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.502+0000 7f2e8affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:21.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.505+0000 7f2ea080d640 10 librados: watch_flush enter 2024-03-29T10:58:21.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 10 librados: watch_flush exit 2024-03-29T10:58:21.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-29T10:58:21.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 10 client.4424.objecter successfully canceled tick 2024-03-29T10:58:21.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 >> v1:172.21.15.22:6816/380221644 conn(0x7f2e7c06dbd0 legacy=0x7f2e7c070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 10 monclient: shutdown 2024-03-29T10:58:21.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:21.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 >> v1:172.21.15.22:6790/0 conn(0x7f2e9c06d9d0 legacy=0x7f2e9c1382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:21.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 shutdown_connections 2024-03-29T10:58:21.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.506+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 >> 172.21.15.22:0/1047989905 conn(0x7f2e9c06b660 msgr2=0x7f2e9c13ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:21.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.507+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 shutdown_connections 2024-03-29T10:58:21.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.507+0000 7f2ea080d640 1 -- 172.21.15.22:0/1047989905 wait complete. 2024-03-29T10:58:21.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:21.507+0000 7f2ea080d640 1 librados: shutdown 2024-03-29T10:58:21.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:28.311 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.479+0000 7fbde90f4640 10 monclient: get_monmap_and_config 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.479+0000 7fbde90f4640 10 monclient: build_initial_monmap 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.479+0000 7fbde90f4640 10 monclient: monmap: 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:28.479940+0000 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:28.479940+0000 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:28.459 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:28.460 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:28.460 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:28.460 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:28.460 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.480+0000 7fbde90f4640 1 Processor -- start 2024-03-29T10:58:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.480+0000 7fbde90f4640 1 -- start start 2024-03-29T10:58:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.480+0000 7fbde90f4640 10 monclient: init 2024-03-29T10:58:28.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 10 monclient(hunting): picked mon.noname-a con 0x7fbde41457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 10 monclient(hunting): picked mon.noname-c con 0x7fbde413f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 10 monclient(hunting): picked mon.noname-b con 0x7fbde40a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde40aaef0 con 0x7fbde41457f0 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde4145bd0 con 0x7fbde413f650 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde4145eb0 con 0x7fbde40a9d70 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:28.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.481+0000 7fbde90f4640 10 monclient(hunting): authenticate will time out at 1009.785355s 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde359e640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fbde40a9d70 0x7fbde414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35688/0 (socket says 172.21.15.22:35688) 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde359e640 1 -- 172.21.15.22:0/1789248725 learned_addr learned my addr 172.21.15.22:0/1789248725 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201463247 0 0) 0x7fbde4145eb0 con 0x7fbde40a9d70 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde1d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde1d9b640 10 monclient(hunting): my global_id is 4427 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.482+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdcc003520 con 0x7fbde40a9d70 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286011363 0 0) 0x7fbde40aaef0 con 0x7fbde41457f0 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:28.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient(hunting): my global_id is 4362 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbde4145eb0 con 0x7fbde41457f0 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 641052775 0 0) 0x7fbde4145eb0 con 0x7fbde41457f0 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 >> v1:172.21.15.22:6790/0 conn(0x7fbde40a9d70 legacy=0x7fbde414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 >> v1:172.21.15.26:6789/0 conn(0x7fbde413f650 legacy=0x7fbde40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 monclient: found mon.noname-a 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde4146340 con 0x7fbde41457f0 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: _finish_auth 0 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbdd40019f0 con 0x7fbde41457f0 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: dump: 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde90f4640 5 monclient: authenticate success, global_id 4362 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde90f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdd4001bd0 con 0x7fbde41457f0 2024-03-29T10:58:28.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 1 -- 172.21.15.22:0/1789248725 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbdd40019f0 con 0x7fbde41457f0 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.484+0000 7fbde1d9b640 10 monclient: dump: 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:28.470 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.485+0000 7fbde90f4640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.485+0000 7fbde90f4640 4 set_mon_vals no callback set 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.485+0000 7fbde90f4640 10 monclient: shutdown 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.485+0000 7fbde90f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.485+0000 7fbde90f4640 1 -- 172.21.15.22:0/1789248725 >> v1:172.21.15.22:6789/0 conn(0x7fbde41457f0 legacy=0x7fbde413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.485+0000 7fbde90f4640 1 -- 172.21.15.22:0/1789248725 shutdown_connections 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.485+0000 7fbde90f4640 1 -- 172.21.15.22:0/1789248725 >> 172.21.15.22:0/1789248725 conn(0x7fbde4066a00 msgr2=0x7fbde4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.486+0000 7fbde90f4640 1 -- 172.21.15.22:0/1789248725 shutdown_connections 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.486+0000 7fbde90f4640 1 -- 172.21.15.22:0/1789248725 wait complete. 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.487+0000 7fbde90f4640 10 monclient: build_initial_monmap 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.487+0000 7fbde90f4640 10 monclient: monmap: 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:28.471 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:28.472 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:28.472 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:28.472 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:28.472 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:28.472 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:28.472 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:28.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.487+0000 7fbde90f4640 1 librados: starting msgr at 2024-03-29T10:58:28.476 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.487+0000 7fbde90f4640 1 librados: starting objecter 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.488+0000 7fbde90f4640 1 Processor -- start 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.488+0000 7fbde90f4640 1 -- start start 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.488+0000 7fbde90f4640 1 librados: setting wanted keys 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.488+0000 7fbde90f4640 1 librados: calling monclient init 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.488+0000 7fbde90f4640 10 monclient: init 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.489+0000 7fbde90f4640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.489+0000 7fbde90f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.489+0000 7fbde90f4640 10 monclient(hunting): picked mon.noname-c con 0x7fbde40a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.489+0000 7fbde2d9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fbde40a9d70 0x7fbde4142650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35818/0 (socket says 172.21.15.22:35818) 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.489+0000 7fbde2d9d640 1 -- 172.21.15.22:0/1733116951 learned_addr learned my addr 172.21.15.22:0/1733116951 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.489+0000 7fbdcb7fe640 10 client.?.objecter ms_handle_connect 0x7fbde40a9d70 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.489+0000 7fbdcb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.490+0000 7fbde90f4640 10 monclient(hunting): picked mon.noname-b con 0x7fbde413f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:28.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.490+0000 7fbde90f4640 10 monclient(hunting): picked mon.noname-a con 0x7fbde41457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:28.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.490+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde4142d60 con 0x7fbde41457f0 2024-03-29T10:58:28.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.490+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde40ac440 con 0x7fbde413f650 2024-03-29T10:58:28.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.490+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbde40ac720 con 0x7fbde40a9d70 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.490+0000 7fbde90f4640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.490+0000 7fbde90f4640 10 monclient(hunting): authenticate will time out at 1009.794433s 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.491+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488824648 0 0) 0x7fbde40ac720 con 0x7fbde40a9d70 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.491+0000 7fbdcb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.491+0000 7fbdcb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.491+0000 7fbdcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.491+0000 7fbdcb7fe640 10 monclient(hunting): my global_id is 4430 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.491+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd0004eb0 con 0x7fbde40a9d70 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3725213118 0 0) 0x7fbdd0004eb0 con 0x7fbde40a9d70 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbde40ac720 con 0x7fbde40a9d70 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbdd40019f0 con 0x7fbde40a9d70 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): dump: 2024-03-29T10:58:28.479 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 client.?.objecter ms_handle_connect 0x7fbde41457f0 2024-03-29T10:58:28.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767726487 0 0) 0x7fbde4142d60 con 0x7fbde41457f0 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient(hunting): my global_id is 4365 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd0004eb0 con 0x7fbde41457f0 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3725255186 0 0) 0x7fbde40ac720 con 0x7fbde40a9d70 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 >> v1:172.21.15.26:6789/0 conn(0x7fbde413f650 legacy=0x7fbde4136f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 >> v1:172.21.15.22:6789/0 conn(0x7fbde41457f0 legacy=0x7fbde41e2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 monclient: found mon.c 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:28.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde4144a20 con 0x7fbde40a9d70 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient: _finish_auth 0 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.493+0000 7fbdcb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 5 monclient: authenticate success, global_id 4430 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 10 monclient: _renew_subs 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbde4144cd0 con 0x7fbde40a9d70 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 10 monclient: _renew_subs 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde413ff80 con 0x7fbde40a9d70 2024-03-29T10:58:28.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.494+0000 7fbde90f4640 1 librados: init done 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdd4003c40 con 0x7fbde40a9d70 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbdcb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbdd4004090 con 0x7fbde40a9d70 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbdcb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbdcb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbdcb7fe640 10 monclient: dump: 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:28.484 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbde90f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.495+0000 7fbde159a640 4 set_mon_vals no callback set 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.499+0000 7fbde90f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.499+0000 7fbde90f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.499+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde4142110 con 0x7fbde40a9d70 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.499+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fbdd40041f0 con 0x7fbde40a9d70 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.500+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fbdd40850c0 con 0x7fbde40a9d70 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.500+0000 7fbdcb7fe640 10 client.4430.objecter ms_dispatch 0x7fbde4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.500+0000 7fbdcb7fe640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.500+0000 7fbdcb7fe640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.500+0000 7fbdcb7fe640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.501+0000 7fbde259c640 10 monclient: get_auth_request con 0x7fbdd006dd60 auth_method 0 2024-03-29T10:58:28.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.501+0000 7fbdcb7fe640 10 client.4430.objecter ms_handle_connect 0x7fbdd006dd60 2024-03-29T10:58:28.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.507+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbdd40509a0 con 0x7fbde40a9d70 2024-03-29T10:58:28.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.508+0000 7fbdcb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:28.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.508+0000 7fbdcb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:28.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.844+0000 7fbde90f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:28.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.844+0000 7fbde90f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:28.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.844+0000 7fbde90f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:28.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.844+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbde4075f10 con 0x7fbde40a9d70 2024-03-29T10:58:28.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.845+0000 7fbdcb7fe640 1 -- 172.21.15.22:0/1733116951 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fbdd40528d0 con 0x7fbde40a9d70 2024-03-29T10:58:28.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.845+0000 7fbdcb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:28.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.845+0000 7fbdcb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:28.826 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:28.826 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:28.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 10 librados: watch_flush enter 2024-03-29T10:58:28.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 10 librados: watch_flush exit 2024-03-29T10:58:28.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T10:58:28.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 10 client.4430.objecter successfully canceled tick 2024-03-29T10:58:28.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 >> v1:172.21.15.22:6816/380221644 conn(0x7fbdd006dd60 legacy=0x7fbdd0070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 10 monclient: shutdown 2024-03-29T10:58:28.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:28.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.849+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 >> v1:172.21.15.22:6790/0 conn(0x7fbde40a9d70 legacy=0x7fbde4142650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:28.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.850+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 shutdown_connections 2024-03-29T10:58:28.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.850+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 >> 172.21.15.22:0/1733116951 conn(0x7fbde4066a00 msgr2=0x7fbde4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:28.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.851+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 shutdown_connections 2024-03-29T10:58:28.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.852+0000 7fbde90f4640 1 -- 172.21.15.22:0/1733116951 wait complete. 2024-03-29T10:58:28.833 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:28.853+0000 7fbde90f4640 1 librados: shutdown 2024-03-29T10:58:28.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:35.659 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.846+0000 7fa332530640 10 monclient: get_monmap_and_config 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.846+0000 7fa332530640 10 monclient: build_initial_monmap 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.846+0000 7fa332530640 10 monclient: monmap: 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:35.848714+0000 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:35.848714+0000 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:35.828 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:35.829 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:35.829 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:35.829 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:35.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.848+0000 7fa332530640 1 Processor -- start 2024-03-29T10:58:35.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.848+0000 7fa332530640 1 -- start start 2024-03-29T10:58:35.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.848+0000 7fa332530640 10 monclient: init 2024-03-29T10:58:35.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.849+0000 7fa332530640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:35.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.849+0000 7fa332530640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:35.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.849+0000 7fa332530640 10 monclient(hunting): picked mon.noname-a con 0x7fa32c1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:35.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.849+0000 7fa332530640 10 monclient(hunting): picked mon.noname-b con 0x7fa32c13f650 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:35.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.849+0000 7fa332530640 10 monclient(hunting): picked mon.noname-c con 0x7fa32c0a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:35.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.850+0000 7fa332530640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c0aaef0 con 0x7fa32c1457f0 2024-03-29T10:58:35.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.850+0000 7fa332530640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c145bd0 con 0x7fa32c0a9d70 2024-03-29T10:58:35.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.850+0000 7fa332530640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c145eb0 con 0x7fa32c13f650 2024-03-29T10:58:35.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.850+0000 7fa332530640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:35.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.850+0000 7fa332530640 10 monclient(hunting): authenticate will time out at 1017.155171s 2024-03-29T10:58:35.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.849+0000 7fa32b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa32c13f650 0x7fa32c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41456/0 (socket says 172.21.15.22:41456) 2024-03-29T10:58:35.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.850+0000 7fa32b7fe640 1 -- 172.21.15.22:0/1219808686 learned_addr learned my addr 172.21.15.22:0/1219808686 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:35.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875668461 0 0) 0x7fa32c145eb0 con 0x7fa32c13f650 2024-03-29T10:58:35.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:35.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:35.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:35.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 10 monclient(hunting): my global_id is 4433 2024-03-29T10:58:35.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa314003520 con 0x7fa32c13f650 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233694520 0 0) 0x7fa314003520 con 0x7fa32c13f650 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 >> v1:172.21.15.26:6789/0 conn(0x7fa32c0a9d70 legacy=0x7fa32c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.854+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 >> v1:172.21.15.22:6789/0 conn(0x7fa32c1457f0 legacy=0x7fa32c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 1 monclient: found mon.noname-b 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c146340 con 0x7fa32c13f650 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 10 monclient: _finish_auth 0 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:35.839 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa31c001690 con 0x7fa32c13f650 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa32affd640 10 monclient: dump: 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:35.840 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa332530640 5 monclient: authenticate success, global_id 4433 2024-03-29T10:58:35.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.857+0000 7fa332530640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:35.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa31c001920 con 0x7fa32c13f650 2024-03-29T10:58:35.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa32affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa32affd640 1 -- 172.21.15.22:0/1219808686 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa31c001cb0 con 0x7fa32c13f650 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa32affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa32affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa32affd640 10 monclient: dump: 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:35.842 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa332530640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.858+0000 7fa332530640 4 set_mon_vals no callback set 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.859+0000 7fa332530640 10 monclient: shutdown 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.859+0000 7fa332530640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.859+0000 7fa332530640 1 -- 172.21.15.22:0/1219808686 >> v1:172.21.15.22:6790/0 conn(0x7fa32c13f650 legacy=0x7fa32c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.860+0000 7fa332530640 1 -- 172.21.15.22:0/1219808686 shutdown_connections 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.860+0000 7fa332530640 1 -- 172.21.15.22:0/1219808686 >> 172.21.15.22:0/1219808686 conn(0x7fa32c066a00 msgr2=0x7fa32c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.860+0000 7fa332530640 1 -- 172.21.15.22:0/1219808686 shutdown_connections 2024-03-29T10:58:35.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.860+0000 7fa332530640 1 -- 172.21.15.22:0/1219808686 wait complete. 2024-03-29T10:58:35.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.861+0000 7fa332530640 10 monclient: build_initial_monmap 2024-03-29T10:58:35.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.861+0000 7fa332530640 10 monclient: monmap: 2024-03-29T10:58:35.845 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:35.845 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:35.845 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:35.845 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:35.845 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.862+0000 7fa332530640 1 librados: starting msgr at 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.862+0000 7fa332530640 1 librados: starting objecter 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.862+0000 7fa332530640 1 Processor -- start 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.862+0000 7fa332530640 1 -- start start 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.862+0000 7fa332530640 1 librados: setting wanted keys 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.862+0000 7fa332530640 1 librados: calling monclient init 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.862+0000 7fa332530640 10 monclient: init 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:35.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 10 monclient(hunting): picked mon.noname-a con 0x7fa32c0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 10 monclient(hunting): picked mon.noname-b con 0x7fa32c13f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 10 monclient(hunting): picked mon.noname-c con 0x7fa32c1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c137ae0 con 0x7fa32c0a9d70 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c06f790 con 0x7fa32c13f650 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32c06fa70 con 0x7fa32c1457f0 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:35.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.863+0000 7fa332530640 10 monclient(hunting): authenticate will time out at 1017.168846s 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.864+0000 7fa330acd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa32c1457f0 0x7fa32c1373d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:41712/0 (socket says 172.21.15.22:41712) 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.864+0000 7fa330acd640 1 -- 172.21.15.22:0/1862405808 learned_addr learned my addr 172.21.15.22:0/1862405808 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 10 client.?.objecter ms_handle_connect 0x7fa32c1457f0 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775748955 0 0) 0x7fa32c06fa70 con 0x7fa32c1457f0 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 10 monclient(hunting): my global_id is 4436 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.865+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa318004eb0 con 0x7fa32c1457f0 2024-03-29T10:58:35.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.868+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2763562323 0 0) 0x7fa318004eb0 con 0x7fa32c1457f0 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.868+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa32c06fa70 con 0x7fa32c1457f0 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.868+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa324001d30 con 0x7fa32c1457f0 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.868+0000 7fa328ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.868+0000 7fa328ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.868+0000 7fa328ff9640 10 monclient(hunting): dump: 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:35.854 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.868+0000 7fa328ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 10 client.?.objecter ms_handle_connect 0x7fa32c13f650 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910237987 0 0) 0x7fa32c06f790 con 0x7fa32c13f650 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 10 monclient(hunting): my global_id is 4357 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.869+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa318004eb0 con 0x7fa32c13f650 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 22012061 0 0) 0x7fa318004eb0 con 0x7fa32c13f650 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa32c06f790 con 0x7fa32c13f650 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa31c0013b0 con 0x7fa32c13f650 2024-03-29T10:58:35.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 10 monclient(hunting): dump: 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 126249184 0 0) 0x7fa32c06fa70 con 0x7fa32c1457f0 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 >> v1:172.21.15.26:6789/0 conn(0x7fa32c13f650 legacy=0x7fa32c136cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 >> v1:172.21.15.22:6789/0 conn(0x7fa32c0a9d70 legacy=0x7fa32c140100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:35.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 monclient: found mon.c 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32c06fd50 con 0x7fa32c1457f0 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 10 monclient: _finish_auth 0 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.870+0000 7fa328ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 5 monclient: authenticate success, global_id 4436 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 10 monclient: _renew_subs 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa32c06e400 con 0x7fa32c1457f0 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 10 monclient: _renew_subs 2024-03-29T10:58:35.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:35.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa32c140810 con 0x7fa32c1457f0 2024-03-29T10:58:35.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.871+0000 7fa332530640 1 librados: init done 2024-03-29T10:58:35.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.873+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa324002030 con 0x7fa32c1457f0 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.873+0000 7fa328ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.873+0000 7fa32a7fc640 4 set_mon_vals no callback set 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.873+0000 7fa332530640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.874+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa324003de0 con 0x7fa32c1457f0 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.874+0000 7fa328ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.874+0000 7fa328ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.874+0000 7fa328ff9640 10 monclient: dump: 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:35.860 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.874+0000 7fa332530640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.875+0000 7fa332530640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.875+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa30c005080 con 0x7fa32c1457f0 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.875+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fa324003f40 con 0x7fa32c1457f0 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.876+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fa324054cb0 con 0x7fa32c1457f0 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.876+0000 7fa328ff9640 10 client.4436.objecter ms_dispatch 0x7fa32c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.876+0000 7fa328ff9640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.876+0000 7fa328ff9640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.876+0000 7fa328ff9640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T10:58:35.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.876+0000 7fa32bfff640 10 monclient: get_auth_request con 0x7fa31806dde0 auth_method 0 2024-03-29T10:58:35.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.877+0000 7fa328ff9640 10 client.4436.objecter ms_handle_connect 0x7fa31806dde0 2024-03-29T10:58:35.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.884+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa324050f50 con 0x7fa32c1457f0 2024-03-29T10:58:35.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.884+0000 7fa328ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:35.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:35.884+0000 7fa328ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:36.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.195+0000 7fa332530640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:36.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.195+0000 7fa332530640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:36.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.195+0000 7fa332530640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:36.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.195+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa30c005330 con 0x7fa32c1457f0 2024-03-29T10:58:36.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.196+0000 7fa328ff9640 1 -- 172.21.15.22:0/1862405808 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fa3240510f0 con 0x7fa32c1457f0 2024-03-29T10:58:36.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.196+0000 7fa328ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:36.177 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.196+0000 7fa328ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:36.178 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:36.178 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:36.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.199+0000 7fa332530640 10 librados: watch_flush enter 2024-03-29T10:58:36.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.199+0000 7fa332530640 10 librados: watch_flush exit 2024-03-29T10:58:36.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.199+0000 7fa332530640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T10:58:36.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.200+0000 7fa332530640 10 client.4436.objecter successfully canceled tick 2024-03-29T10:58:36.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.200+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 >> v1:172.21.15.22:6816/380221644 conn(0x7fa31806dde0 legacy=0x7fa3180702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:36.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.200+0000 7fa332530640 10 monclient: shutdown 2024-03-29T10:58:36.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.200+0000 7fa332530640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:36.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.200+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 >> v1:172.21.15.22:6790/0 conn(0x7fa32c1457f0 legacy=0x7fa32c1373d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:36.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.201+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 shutdown_connections 2024-03-29T10:58:36.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.201+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 >> 172.21.15.22:0/1862405808 conn(0x7fa32c066a00 msgr2=0x7fa32c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:36.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.201+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 shutdown_connections 2024-03-29T10:58:36.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.201+0000 7fa332530640 1 -- 172.21.15.22:0/1862405808 wait complete. 2024-03-29T10:58:36.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:36.201+0000 7fa332530640 1 librados: shutdown 2024-03-29T10:58:36.198 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:43.004 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:43.171 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.189+0000 7ffbab117640 10 monclient: get_monmap_and_config 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.189+0000 7ffbab117640 10 monclient: build_initial_monmap 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.189+0000 7ffbab117640 10 monclient: monmap: 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:43.191432+0000 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:43.191432+0000 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:43.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.190+0000 7ffbab117640 1 Processor -- start 2024-03-29T10:58:43.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.190+0000 7ffbab117640 1 -- start start 2024-03-29T10:58:43.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.190+0000 7ffbab117640 10 monclient: init 2024-03-29T10:58:43.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.191+0000 7ffbab117640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:43.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.191+0000 7ffbab117640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:43.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 10 monclient(hunting): picked mon.noname-b con 0x7ffba4071e60 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:43.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 10 monclient(hunting): picked mon.noname-c con 0x7ffba40a9c70 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:43.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 10 monclient(hunting): picked mon.noname-a con 0x7ffba413f7e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:43.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40731b0 con 0x7ffba413f7e0 2024-03-29T10:58:43.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba40aaeb0 con 0x7ffba40a9c70 2024-03-29T10:58:43.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba4072c90 con 0x7ffba4071e60 2024-03-29T10:58:43.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:43.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.192+0000 7ffbab117640 10 monclient(hunting): authenticate will time out at 1024.497178s 2024-03-29T10:58:43.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.193+0000 7ffbaa115640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ffba4071e60 0x7ffba413f0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47572/0 (socket says 172.21.15.22:47572) 2024-03-29T10:58:43.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.193+0000 7ffbaa115640 1 -- 172.21.15.22:0/531567842 learned_addr learned my addr 172.21.15.22:0/531567842 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075259413 0 0) 0x7ffba4072c90 con 0x7ffba4071e60 2024-03-29T10:58:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): _negotiate 2024-03-29T10:58:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:43.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): my global_id is 4439 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb90003520 con 0x7ffba4071e60 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 841513223 0 0) 0x7ffba40aaeb0 con 0x7ffba40a9c70 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): _negotiate 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient(hunting): my global_id is 4360 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba4072c90 con 0x7ffba40a9c70 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 18652475 0 0) 0x7ffb90003520 con 0x7ffba4071e60 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 >> v1:172.21.15.26:6789/0 conn(0x7ffba40a9c70 legacy=0x7ffba41453b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 >> v1:172.21.15.22:6789/0 conn(0x7ffba413f7e0 legacy=0x7ffba414d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient: _finish_hunting 0 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 monclient: found mon.noname-b 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:58:43.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba4145ac0 con 0x7ffba4071e60 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient: _finish_auth 0 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 10 monclient: _check_auth_tickets 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.196+0000 7ffba9113640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.197+0000 7ffbab117640 5 monclient: authenticate success, global_id 4439 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.197+0000 7ffbab117640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.197+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb94001690 con 0x7ffba4071e60 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.197+0000 7ffba9113640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.197+0000 7ffba9113640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.197+0000 7ffba9113640 10 monclient: dump: 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:43.181 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:43.182 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:43.182 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:43.182 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:43.182 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.197+0000 7ffbab117640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.198+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb94001870 con 0x7ffba4071e60 2024-03-29T10:58:43.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.198+0000 7ffba9113640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.198+0000 7ffbab117640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.198+0000 7ffbab117640 4 set_mon_vals no callback set 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.199+0000 7ffbab117640 10 monclient: shutdown 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.199+0000 7ffbab117640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.199+0000 7ffbab117640 1 -- 172.21.15.22:0/531567842 >> v1:172.21.15.22:6790/0 conn(0x7ffba4071e60 legacy=0x7ffba413f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.199+0000 7ffba9113640 1 -- 172.21.15.22:0/531567842 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb94001690 con 0x7ffba4071e60 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.200+0000 7ffba9113640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.200+0000 7ffbab117640 1 -- 172.21.15.22:0/531567842 shutdown_connections 2024-03-29T10:58:43.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.200+0000 7ffbab117640 1 -- 172.21.15.22:0/531567842 >> 172.21.15.22:0/531567842 conn(0x7ffba40b3a30 msgr2=0x7ffba40b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.200+0000 7ffbab117640 1 -- 172.21.15.22:0/531567842 shutdown_connections 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.200+0000 7ffbab117640 1 -- 172.21.15.22:0/531567842 wait complete. 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.201+0000 7ffbab117640 10 monclient: build_initial_monmap 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.201+0000 7ffbab117640 10 monclient: monmap: 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:43.186 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.202+0000 7ffbab117640 1 librados: starting msgr at 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.202+0000 7ffbab117640 1 librados: starting objecter 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.202+0000 7ffbab117640 1 Processor -- start 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.202+0000 7ffbab117640 1 -- start start 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.202+0000 7ffbab117640 1 librados: setting wanted keys 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.202+0000 7ffbab117640 1 librados: calling monclient init 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.202+0000 7ffbab117640 10 monclient: init 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 10 monclient(hunting): picked mon.noname-b con 0x7ffba4071e60 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 10 monclient(hunting): picked mon.noname-c con 0x7ffba40a9c70 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 10 monclient(hunting): picked mon.noname-a con 0x7ffba413f7e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba41448e0 con 0x7ffba413f7e0 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba4144620 con 0x7ffba4071e60 2024-03-29T10:58:43.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffba411b810 con 0x7ffba40a9c70 2024-03-29T10:58:43.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:43.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.203+0000 7ffbab117640 10 monclient(hunting): authenticate will time out at 1024.508330s 2024-03-29T10:58:43.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.206+0000 7ffba9914640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ffba40a9c70 0x7ffba41d6d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:47776/0 (socket says 172.21.15.22:47776) 2024-03-29T10:58:43.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.206+0000 7ffba9914640 1 -- 172.21.15.22:0/3698998970 learned_addr learned my addr 172.21.15.22:0/3698998970 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:43.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.206+0000 7ffb9affd640 10 client.?.objecter ms_handle_connect 0x7ffba40a9c70 2024-03-29T10:58:43.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.206+0000 7ffb9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:43.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.207+0000 7ffb9affd640 10 client.?.objecter ms_handle_connect 0x7ffba4071e60 2024-03-29T10:58:43.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.207+0000 7ffb9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:43.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194294981 0 0) 0x7ffba4144620 con 0x7ffba4071e60 2024-03-29T10:58:43.199 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): my global_id is 4363 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb88004eb0 con 0x7ffba4071e60 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 client.?.objecter ms_handle_connect 0x7ffba413f7e0 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371849478 0 0) 0x7ffba41448e0 con 0x7ffba413f7e0 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): my global_id is 4368 2024-03-29T10:58:43.200 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba4144620 con 0x7ffba413f7e0 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798089620 0 0) 0x7ffba411b810 con 0x7ffba40a9c70 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 10 monclient(hunting): my global_id is 4442 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.208+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffba41448e0 con 0x7ffba40a9c70 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 630709814 0 0) 0x7ffba4144620 con 0x7ffba413f7e0 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffba411b810 con 0x7ffba413f7e0 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffba00016f0 con 0x7ffba413f7e0 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 10 monclient(hunting): dump: 2024-03-29T10:58:43.201 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1824917142 0 0) 0x7ffba41448e0 con 0x7ffba40a9c70 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffba4144620 con 0x7ffba40a9c70 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb9c001cb0 con 0x7ffba40a9c70 2024-03-29T10:58:43.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.209+0000 7ffb9affd640 10 monclient(hunting): dump: 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.210+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422886354 0 0) 0x7ffb88004eb0 con 0x7ffba4071e60 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.210+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffba41448e0 con 0x7ffba4071e60 2024-03-29T10:58:43.203 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.210+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb94001d60 con 0x7ffba4071e60 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.210+0000 7ffb9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.210+0000 7ffb9affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.210+0000 7ffb9affd640 10 monclient(hunting): dump: 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2856840383 0 0) 0x7ffba4144620 con 0x7ffba40a9c70 2024-03-29T10:58:43.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 >> v1:172.21.15.26:6789/0 conn(0x7ffba4071e60 legacy=0x7ffba4143f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 >> v1:172.21.15.22:6789/0 conn(0x7ffba413f7e0 legacy=0x7ffba41d7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 1 monclient: found mon.c 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba41434a0 con 0x7ffba40a9c70 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: _finish_auth 0 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 5 monclient: authenticate success, global_id 4442 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 10 monclient: _renew_subs 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:43.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffba4139230 con 0x7ffba40a9c70 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 10 monclient: _renew_subs 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffba41e29d0 con 0x7ffba40a9c70 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffbab117640 1 librados: init done 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb9c003990 con 0x7ffba40a9c70 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffb9c003da0 con 0x7ffba40a9c70 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffb9affd640 10 monclient: dump: 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:43.206 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.211+0000 7ffba8912640 4 set_mon_vals no callback set 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.213+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7ffb9c003f00 con 0x7ffba40a9c70 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.213+0000 7ffbab117640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.213+0000 7ffbab117640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.213+0000 7ffbab117640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:43.207 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.213+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c005080 con 0x7ffba40a9c70 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.214+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7ffb9c084d20 con 0x7ffba40a9c70 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.214+0000 7ffb9affd640 10 client.4442.objecter ms_dispatch 0x7ffba4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.214+0000 7ffb9affd640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.214+0000 7ffb9affd640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.214+0000 7ffb9affd640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.214+0000 7ffbaa115640 10 monclient: get_auth_request con 0x7ffb8806e210 auth_method 0 2024-03-29T10:58:43.208 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.214+0000 7ffb9affd640 10 client.4442.objecter ms_handle_connect 0x7ffb8806e210 2024-03-29T10:58:43.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.226+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb9c050600 con 0x7ffba40a9c70 2024-03-29T10:58:43.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.226+0000 7ffb9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:43.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.226+0000 7ffb9affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.583+0000 7ffbab117640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.583+0000 7ffbab117640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:43.568 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:43.568 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.583+0000 7ffbab117640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.583+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb8c005330 con 0x7ffba40a9c70 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.585+0000 7ffb9affd640 1 -- 172.21.15.22:0/3698998970 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7ffb9c052530 con 0x7ffba40a9c70 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.585+0000 7ffb9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.585+0000 7ffb9affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.590+0000 7ffbab117640 10 librados: watch_flush enter 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.590+0000 7ffbab117640 10 librados: watch_flush exit 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.590+0000 7ffbab117640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.590+0000 7ffbab117640 10 client.4442.objecter successfully canceled tick 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.590+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 >> v1:172.21.15.22:6816/380221644 conn(0x7ffb8806e210 legacy=0x7ffb880706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.591+0000 7ffbab117640 10 monclient: shutdown 2024-03-29T10:58:43.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.591+0000 7ffbab117640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.591+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 >> v1:172.21.15.22:6790/0 conn(0x7ffba40a9c70 legacy=0x7ffba41d6d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.592+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 shutdown_connections 2024-03-29T10:58:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.592+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 >> 172.21.15.22:0/3698998970 conn(0x7ffba40b3a30 msgr2=0x7ffba40b15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.594+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 shutdown_connections 2024-03-29T10:58:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.594+0000 7ffbab117640 1 -- 172.21.15.22:0/3698998970 wait complete. 2024-03-29T10:58:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:43.594+0000 7ffbab117640 1 librados: shutdown 2024-03-29T10:58:43.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:50.405 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:50.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.563+0000 7f1bd6fc5640 10 monclient: get_monmap_and_config 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.563+0000 7f1bd6fc5640 10 monclient: build_initial_monmap 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.563+0000 7f1bd6fc5640 10 monclient: monmap: 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:50.564675+0000 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:50.564675+0000 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:50.544 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:50.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.566+0000 7f1bd6fc5640 1 Processor -- start 2024-03-29T10:58:50.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.566+0000 7f1bd6fc5640 1 -- start start 2024-03-29T10:58:50.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.566+0000 7f1bd6fc5640 10 monclient: init 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc8038c60 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc80324b0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc80d11e0 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80caf80 con 0x7f1bc80324b0 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc8039850 con 0x7f1bc80d11e0 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80c78d0 con 0x7f1bc8038c60 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd6fc5640 10 monclient(hunting): authenticate will time out at 1031.871686s 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd57c2640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f1bc80324b0 0x7f1bc80cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:41100/0 (socket says 172.21.15.22:41100) 2024-03-29T10:58:50.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.567+0000 7f1bd57c2640 1 -- 172.21.15.22:0/2156812299 learned_addr learned my addr 172.21.15.22:0/2156812299 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401705842 0 0) 0x7f1bc80c78d0 con 0x7f1bc8038c60 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient(hunting): my global_id is 4445 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bbc003520 con 0x7f1bc8038c60 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465783553 0 0) 0x7f1bbc003520 con 0x7f1bc8038c60 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 >> v1:172.21.15.26:6789/0 conn(0x7f1bc80d11e0 legacy=0x7f1bc80d35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 >> v1:172.21.15.22:6789/0 conn(0x7f1bc80324b0 legacy=0x7f1bc80cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: _finish_hunting 0 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 monclient: found mon.noname-b 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:50.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc80c7a60 con 0x7f1bc8038c60 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: _finish_auth 0 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: _check_auth_tickets 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1bcc001690 con 0x7f1bc8038c60 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: dump: 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:50.551 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd6fc5640 5 monclient: authenticate success, global_id 4445 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd6fc5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bcc001860 con 0x7f1bc8038c60 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 1 -- 172.21.15.22:0/2156812299 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1bcc001cb0 con 0x7f1bc8038c60 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd4fc1640 10 monclient: dump: 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.552 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.553 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:50.553 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:50.553 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:50.553 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:50.553 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:50.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd6fc5640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:50.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.568+0000 7f1bd6fc5640 4 set_mon_vals no callback set 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.569+0000 7f1bd6fc5640 10 monclient: shutdown 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.569+0000 7f1bd6fc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.569+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/2156812299 >> v1:172.21.15.22:6790/0 conn(0x7f1bc8038c60 legacy=0x7f1bc80c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.569+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/2156812299 shutdown_connections 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.569+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/2156812299 >> 172.21.15.22:0/2156812299 conn(0x7f1bc8005c20 msgr2=0x7f1bc8006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.570+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/2156812299 shutdown_connections 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.570+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/2156812299 wait complete. 2024-03-29T10:58:50.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.571+0000 7f1bd6fc5640 10 monclient: build_initial_monmap 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.571+0000 7f1bd6fc5640 10 monclient: monmap: 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.572+0000 7f1bd6fc5640 1 librados: starting msgr at 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.572+0000 7f1bd6fc5640 1 librados: starting objecter 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.572+0000 7f1bd6fc5640 1 Processor -- start 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.572+0000 7f1bd6fc5640 1 -- start start 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.572+0000 7f1bd6fc5640 1 librados: setting wanted keys 2024-03-29T10:58:50.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.572+0000 7f1bd6fc5640 1 librados: calling monclient init 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.572+0000 7f1bd6fc5640 10 monclient: init 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.573+0000 7f1bd6fc5640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.573+0000 7f1bd6fc5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.573+0000 7f1bd6fc5640 10 monclient(hunting): picked mon.noname-a con 0x7f1bc80324b0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.573+0000 7f1bd5fc3640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f1bc80324b0 0x7f1bc815c900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:41314/0 (socket says 172.21.15.22:41314) 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.573+0000 7f1bd5fc3640 1 -- 172.21.15.22:0/1120193683 learned_addr learned my addr 172.21.15.22:0/1120193683 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.574+0000 7f1bd6fc5640 10 monclient(hunting): picked mon.noname-c con 0x7f1bc8038c60 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.574+0000 7f1bc67fc640 10 client.?.objecter ms_handle_connect 0x7f1bc80324b0 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.574+0000 7f1bc67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.574+0000 7f1bc67fc640 10 client.?.objecter ms_handle_connect 0x7f1bc8038c60 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.574+0000 7f1bc67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.574+0000 7f1bd6fc5640 10 monclient(hunting): picked mon.noname-b con 0x7f1bc80d11e0 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:50.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.574+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80347a0 con 0x7f1bc80324b0 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80c7800 con 0x7f1bc80d11e0 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bc80c6b40 con 0x7f1bc8038c60 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bd6fc5640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bd6fc5640 10 monclient(hunting): authenticate will time out at 1031.879146s 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3101710622 0 0) 0x7f1bc80c6b40 con 0x7f1bc8038c60 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): my global_id is 4448 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bb4004eb0 con 0x7f1bc8038c60 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1454335411 0 0) 0x7f1bc80347a0 con 0x7f1bc80324b0 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): _negotiate 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 10 monclient(hunting): my global_id is 4374 2024-03-29T10:58:50.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.575+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bc80c6b40 con 0x7f1bc80324b0 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2306915602 0 0) 0x7f1bc80c6b40 con 0x7f1bc80324b0 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bc80347a0 con 0x7f1bc80324b0 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1bcc001c30 con 0x7f1bc80324b0 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 10 monclient(hunting): dump: 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:50.560 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:50.561 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2804098855 0 0) 0x7f1bc80347a0 con 0x7f1bc80324b0 2024-03-29T10:58:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 >> v1:172.21.15.22:6790/0 conn(0x7f1bc8038c60 legacy=0x7f1bc815d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.576+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 >> v1:172.21.15.26:6789/0 conn(0x7f1bc80d11e0 legacy=0x7f1bc81672b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:58:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: _finish_hunting 0 2024-03-29T10:58:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 1 monclient: found mon.a 2024-03-29T10:58:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:58:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bc8034430 con 0x7f1bc80324b0 2024-03-29T10:58:50.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: _finish_auth 0 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: _check_auth_tickets 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 5 monclient: authenticate success, global_id 4374 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 10 monclient: _renew_subs 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bc80c6df0 con 0x7f1bc80324b0 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 10 monclient: _renew_subs 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bc80c0830 con 0x7f1bc80324b0 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bd6fc5640 1 librados: init done 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1bcc003940 con 0x7f1bc80324b0 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1bcc003d10 con 0x7f1bc80324b0 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:50.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc67fc640 10 monclient: dump: 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.577+0000 7f1bc7fff640 4 set_mon_vals no callback set 2024-03-29T10:58:50.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.579+0000 7f1bd6fc5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.579+0000 7f1bd6fc5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.579+0000 7f1bd6fc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.579+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bb8005080 con 0x7f1bc80324b0 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.580+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f1bcc003e70 con 0x7f1bc80324b0 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.581+0000 7f1bd57c2640 10 monclient: get_auth_request con 0x7f1bb406dd30 auth_method 0 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.581+0000 7f1bc67fc640 10 client.4374.objecter ms_handle_connect 0x7f1bb406dd30 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.581+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f1bcc084ae0 con 0x7f1bc80324b0 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.581+0000 7f1bc67fc640 10 client.4374.objecter ms_dispatch 0x7f1bd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.581+0000 7f1bc67fc640 3 client.4374.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.581+0000 7f1bc67fc640 3 client.4374.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:50.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.581+0000 7f1bc67fc640 20 client.4374.objecter dump_active .. 0 homeless 2024-03-29T10:58:50.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.587+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1bcc0503c0 con 0x7f1bc80324b0 2024-03-29T10:58:50.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.587+0000 7f1bc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:50.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.587+0000 7f1bc67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.920+0000 7f1bd6fc5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.920+0000 7f1bd6fc5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.920+0000 7f1bd6fc5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:58:50.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.920+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bb8005330 con 0x7f1bc80324b0 2024-03-29T10:58:50.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.921+0000 7f1bc67fc640 1 -- 172.21.15.22:0/1120193683 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f1bcc0522f0 con 0x7f1bc80324b0 2024-03-29T10:58:50.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.921+0000 7f1bc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:50.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.921+0000 7f1bc67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:50.902 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:50.902 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:50.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 10 librados: watch_flush enter 2024-03-29T10:58:50.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 10 librados: watch_flush exit 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 20 client.4374.objecter shutdown clearing up homeless session... 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 10 client.4374.objecter successfully canceled tick 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 >> v1:172.21.15.22:6816/380221644 conn(0x7f1bb406dd30 legacy=0x7f1bb40701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 10 monclient: shutdown 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 >> v1:172.21.15.22:6789/0 conn(0x7f1bc80324b0 legacy=0x7f1bc815c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 shutdown_connections 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.925+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 >> 172.21.15.22:0/1120193683 conn(0x7f1bc8005c20 msgr2=0x7f1bc803bd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.926+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 shutdown_connections 2024-03-29T10:58:50.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.926+0000 7f1bd6fc5640 1 -- 172.21.15.22:0/1120193683 wait complete. 2024-03-29T10:58:50.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:50.926+0000 7f1bd6fc5640 1 librados: shutdown 2024-03-29T10:58:50.922 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:58:57.729 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:58:57.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.902+0000 7f7418e65640 10 monclient: get_monmap_and_config 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.902+0000 7f7418e65640 10 monclient: build_initial_monmap 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.902+0000 7f7418e65640 10 monclient: monmap: 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:58:57.903205+0000 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:58:57.903205+0000 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:58:57.883 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.903+0000 7f7418e65640 1 Processor -- start 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.903+0000 7f7418e65640 1 -- start start 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.903+0000 7f7418e65640 10 monclient: init 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 10 monclient(hunting): picked mon.noname-c con 0x7f7414071c30 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 10 monclient(hunting): picked mon.noname-b con 0x7f74140a9750 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:57.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 10 monclient(hunting): picked mon.noname-a con 0x7f741413e1e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:57.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74140ab300 con 0x7f741413e1e0 2024-03-29T10:58:57.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74140ac2f0 con 0x7f7414071c30 2024-03-29T10:58:57.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74140ad8d0 con 0x7f74140a9750 2024-03-29T10:58:57.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:57.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7418e65640 10 monclient(hunting): authenticate will time out at 1039.208513s 2024-03-29T10:58:57.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7412d9d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f741413e1e0 0x7f74140ac710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38050/0 (socket says 172.21.15.22:38050) 2024-03-29T10:58:57.885 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.904+0000 7f7412d9d640 1 -- 172.21.15.22:0/522420727 learned_addr learned my addr 172.21.15.22:0/522420727 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:57.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100221476 0 0) 0x7f74140ab300 con 0x7f741413e1e0 2024-03-29T10:58:57.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient(hunting): _negotiate 2024-03-29T10:58:57.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:57.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:57.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient(hunting): my global_id is 4377 2024-03-29T10:58:57.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73fc003520 con 0x7f741413e1e0 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1670759509 0 0) 0x7f73fc003520 con 0x7f741413e1e0 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 >> v1:172.21.15.22:6790/0 conn(0x7f74140a9750 legacy=0x7f741413dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 >> v1:172.21.15.26:6789/0 conn(0x7f7414071c30 legacy=0x7f74140b0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient: _finish_hunting 0 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 1 monclient: found mon.noname-a 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74140ada60 con 0x7f741413e1e0 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient: _finish_auth 0 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 10 monclient: _check_auth_tickets 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.905+0000 7f741159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f740c001a10 con 0x7f741413e1e0 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:57.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 10 monclient: dump: 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:57.896 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f7418e65640 5 monclient: authenticate success, global_id 4377 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f7418e65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f740c001b70 con 0x7f741413e1e0 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f7418e65640 10 monclient: get_monmap_and_config success 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f7418e65640 4 set_mon_vals no callback set 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 1 -- 172.21.15.22:0/522420727 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f740c001f80 con 0x7f741413e1e0 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.906+0000 7f741159a640 10 monclient: dump: 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.897 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.907+0000 7f7418e65640 10 monclient: shutdown 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.907+0000 7f7418e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.907+0000 7f7418e65640 1 -- 172.21.15.22:0/522420727 >> v1:172.21.15.22:6789/0 conn(0x7f741413e1e0 legacy=0x7f74140ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.907+0000 7f7418e65640 1 -- 172.21.15.22:0/522420727 shutdown_connections 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.907+0000 7f7418e65640 1 -- 172.21.15.22:0/522420727 >> 172.21.15.22:0/522420727 conn(0x7f741406b660 msgr2=0x7f741406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.907+0000 7f7418e65640 1 -- 172.21.15.22:0/522420727 shutdown_connections 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.907+0000 7f7418e65640 1 -- 172.21.15.22:0/522420727 wait complete. 2024-03-29T10:58:57.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.908+0000 7f7418e65640 10 monclient: build_initial_monmap 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.908+0000 7f7418e65640 10 monclient: monmap: 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.909+0000 7f7418e65640 1 librados: starting msgr at 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.909+0000 7f7418e65640 1 librados: starting objecter 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.909+0000 7f7418e65640 1 Processor -- start 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.909+0000 7f7418e65640 1 -- start start 2024-03-29T10:58:57.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.909+0000 7f7418e65640 1 librados: setting wanted keys 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.909+0000 7f7418e65640 1 librados: calling monclient init 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.909+0000 7f7418e65640 10 monclient: init 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 10 monclient: _reopen_session rank -1 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 10 monclient(hunting): picked mon.noname-c con 0x7f7414071c30 addr v1:172.21.15.22:6790/0 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 10 monclient(hunting): picked mon.noname-b con 0x7f74140a9750 addr v1:172.21.15.26:6789/0 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 10 monclient(hunting): picked mon.noname-a con 0x7f741413e1e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7414070320 con 0x7f741413e1e0 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74140714a0 con 0x7f74140a9750 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f741406ee40 con 0x7f7414071c30 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 10 monclient(hunting): _renew_subs 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7418e65640 10 monclient(hunting): authenticate will time out at 1039.214551s 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f741259c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7414071c30 0x7f741406d9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52980/0 (socket says 172.21.15.22:52980) 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f741259c640 1 -- 172.21.15.22:0/933167450 learned_addr learned my addr 172.21.15.22:0/933167450 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7402ffd640 10 client.?.objecter ms_handle_connect 0x7f7414071c30 2024-03-29T10:58:57.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.910+0000 7f7402ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 client.?.objecter ms_handle_connect 0x7f741413e1e0 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806059401 0 0) 0x7f741406ee40 con 0x7f7414071c30 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): my global_id is 4454 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73f4004eb0 con 0x7f7414071c30 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836010878 0 0) 0x7f7414070320 con 0x7f741413e1e0 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): _negotiate 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): my global_id is 4380 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f741406ee40 con 0x7f741413e1e0 2024-03-29T10:58:57.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2008834626 0 0) 0x7f73f4004eb0 con 0x7f7414071c30 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7414070320 con 0x7f7414071c30 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7404001a30 con 0x7f7414071c30 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:57.902 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.911+0000 7f7402ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 client.?.objecter ms_handle_connect 0x7f74140a9750 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1963365476 0 0) 0x7f741406ee40 con 0x7f741413e1e0 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73f4004eb0 con 0x7f741413e1e0 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f740c001e80 con 0x7f741413e1e0 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 monclient(hunting): dump: 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.903 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3616131339 0 0) 0x7f7414070320 con 0x7f7414071c30 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 >> v1:172.21.15.26:6789/0 conn(0x7f74140a9750 legacy=0x7f7414136b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 >> v1:172.21.15.22:6789/0 conn(0x7f741413e1e0 legacy=0x7f7414137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 monclient: _finish_hunting 0 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 monclient: found mon.c 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:57.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741406fe00 con 0x7f7414071c30 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 monclient: _finish_auth 0 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 10 monclient: _check_auth_tickets 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7402ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 5 monclient: authenticate success, global_id 4454 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 10 monclient: _renew_subs 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74141380f0 con 0x7f7414071c30 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 10 monclient: _renew_subs 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f741406f970 con 0x7f7414071c30 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.912+0000 7f7418e65640 1 librados: init done 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.913+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7404001c70 con 0x7f7414071c30 2024-03-29T10:58:57.905 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.913+0000 7f7402ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.913+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7404003a60 con 0x7f7414071c30 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.913+0000 7f7402ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.913+0000 7f7402ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.913+0000 7f7402ffd640 10 monclient: dump: 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:58:57.906 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.913+0000 7f7410d99640 4 set_mon_vals no callback set 2024-03-29T10:58:57.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.914+0000 7f7418e65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.914+0000 7f7418e65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.914+0000 7f7418e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.914+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f8005080 con 0x7f7414071c30 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.915+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f7404003c40 con 0x7f7414071c30 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.916+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f74040848d0 con 0x7f7414071c30 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.916+0000 7f7402ffd640 10 client.4454.objecter ms_dispatch 0x7f7414000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.916+0000 7f7402ffd640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.916+0000 7f7402ffd640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.916+0000 7f7402ffd640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.916+0000 7f7411d9b640 10 monclient: get_auth_request con 0x7f73f406dfb0 auth_method 0 2024-03-29T10:58:57.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.916+0000 7f7402ffd640 10 client.4454.objecter ms_handle_connect 0x7f73f406dfb0 2024-03-29T10:58:57.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.923+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74040502f0 con 0x7f7414071c30 2024-03-29T10:58:57.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.923+0000 7f7402ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:58:57.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:57.923+0000 7f7402ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:58:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.286+0000 7f7418e65640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:58.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.286+0000 7f7418e65640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.286+0000 7f7418e65640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:58:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.286+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73f8005330 con 0x7f7414071c30 2024-03-29T10:58:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.286+0000 7f7402ffd640 1 -- 172.21.15.22:0/933167450 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f74040038c0 con 0x7f7414071c30 2024-03-29T10:58:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.286+0000 7f7402ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:58:58.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.286+0000 7f7402ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:58:58.267 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:58:58.267 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 10 librados: watch_flush enter 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 10 librados: watch_flush exit 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 10 client.4454.objecter successfully canceled tick 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 >> v1:172.21.15.22:6816/380221644 conn(0x7f73f406dfb0 legacy=0x7f73f4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 10 monclient: shutdown 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.289+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 >> v1:172.21.15.22:6790/0 conn(0x7f7414071c30 legacy=0x7f741406d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:58:58.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.290+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 shutdown_connections 2024-03-29T10:58:58.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.290+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 >> 172.21.15.22:0/933167450 conn(0x7f741406b660 msgr2=0x7f74140b04d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:58:58.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.290+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 shutdown_connections 2024-03-29T10:58:58.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.290+0000 7f7418e65640 1 -- 172.21.15.22:0/933167450 wait complete. 2024-03-29T10:58:58.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:58:58.290+0000 7f7418e65640 1 librados: shutdown 2024-03-29T10:58:58.284 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:05.090 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:05.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.257+0000 7f9bf9b54640 10 monclient: get_monmap_and_config 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.257+0000 7f9bf9b54640 10 monclient: build_initial_monmap 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.257+0000 7f9bf9b54640 10 monclient: monmap: 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:05.258487+0000 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:05.258487+0000 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:05.238 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:05.239 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:05.239 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:05.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.258+0000 7f9bf9b54640 1 Processor -- start 2024-03-29T10:59:05.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.258+0000 7f9bf9b54640 1 -- start start 2024-03-29T10:59:05.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.258+0000 7f9bf9b54640 10 monclient: init 2024-03-29T10:59:05.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.259+0000 7f9bf9b54640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.259+0000 7f9bf9b54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.259+0000 7f9bf9b54640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf41457f0 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.259+0000 7f9bf9b54640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf413f650 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.259+0000 7f9bf9b54640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf40a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.259+0000 7f9bf3fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f9bf40a9d70 0x7f9bf414d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57614/0 (socket says 172.21.15.22:57614) 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.259+0000 7f9bf3fff640 1 -- 172.21.15.22:0/2617881819 learned_addr learned my addr 172.21.15.22:0/2617881819 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf40aaef0 con 0x7f9bf413f650 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4145bd0 con 0x7f9bf41457f0 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4145eb0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf9b54640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf9b54640 10 monclient(hunting): authenticate will time out at 1046.564646s 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195768561 0 0) 0x7f9bf4145bd0 con 0x7f9bf41457f0 2024-03-29T10:59:05.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf27fc640 10 monclient(hunting): my global_id is 4372 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.260+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bdc003520 con 0x7f9bf41457f0 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321626025 0 0) 0x7f9bf4145eb0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 10 monclient(hunting): my global_id is 4457 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4145bd0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1551096546 0 0) 0x7f9bf4145bd0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 >> v1:172.21.15.26:6789/0 conn(0x7f9bf41457f0 legacy=0x7f9bf413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:05.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.261+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 >> v1:172.21.15.22:6789/0 conn(0x7f9bf413f650 legacy=0x7f9bf40b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 1 monclient: found mon.noname-b 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4146340 con 0x7f9bf40a9d70 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9bec0019f0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf27fc640 10 monclient: dump: 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:05.246 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf9b54640 5 monclient: authenticate success, global_id 4457 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.262+0000 7f9bf9b54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9bec001c10 con 0x7f9bf40a9d70 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf27fc640 1 -- 172.21.15.22:0/2617881819 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9bec001fa0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf27fc640 10 monclient: dump: 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf9b54640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:05.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.263+0000 7f9bf9b54640 4 set_mon_vals no callback set 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.264+0000 7f9bf9b54640 10 monclient: shutdown 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.264+0000 7f9bf9b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.264+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 >> v1:172.21.15.22:6790/0 conn(0x7f9bf40a9d70 legacy=0x7f9bf414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.265+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 shutdown_connections 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.265+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 >> 172.21.15.22:0/2617881819 conn(0x7f9bf4066a00 msgr2=0x7f9bf4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.265+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 shutdown_connections 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.265+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2617881819 wait complete. 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.266+0000 7f9bf9b54640 10 monclient: build_initial_monmap 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.266+0000 7f9bf9b54640 10 monclient: monmap: 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:05.258 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.266+0000 7f9bf9b54640 1 librados: starting msgr at 2024-03-29T10:59:05.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.266+0000 7f9bf9b54640 1 librados: starting objecter 2024-03-29T10:59:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.266+0000 7f9bf9b54640 1 Processor -- start 2024-03-29T10:59:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.267+0000 7f9bf9b54640 1 -- start start 2024-03-29T10:59:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.267+0000 7f9bf9b54640 1 librados: setting wanted keys 2024-03-29T10:59:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.267+0000 7f9bf9b54640 1 librados: calling monclient init 2024-03-29T10:59:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.267+0000 7f9bf9b54640 10 monclient: init 2024-03-29T10:59:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.267+0000 7f9bf9b54640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:05.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.267+0000 7f9bf9b54640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 10 monclient(hunting): picked mon.noname-b con 0x7f9bf40a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 10 monclient(hunting): picked mon.noname-c con 0x7f9bf413f650 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 10 monclient(hunting): picked mon.noname-a con 0x7f9bf41457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4133970 con 0x7f9bf41457f0 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4133e90 con 0x7f9bf40a9d70 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4134020 con 0x7f9bf413f650 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf9b54640 10 monclient(hunting): authenticate will time out at 1046.572209s 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf2ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f9bf413f650 0x7f9bf4134370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57810/0 (socket says 172.21.15.22:57810) 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bf2ffd640 1 -- 172.21.15.22:0/2200692727 learned_addr learned my addr 172.21.15.22:0/2200692727 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 client.?.objecter ms_handle_connect 0x7f9bf413f650 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:05.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 client.?.objecter ms_handle_connect 0x7f9bf40a9d70 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604444356 0 0) 0x7f9bf4134020 con 0x7f9bf413f650 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): my global_id is 4460 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be0004eb0 con 0x7f9bf413f650 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178232202 0 0) 0x7f9bf4133e90 con 0x7f9bf40a9d70 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 10 monclient(hunting): my global_id is 4375 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.268+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bf4134020 con 0x7f9bf40a9d70 2024-03-29T10:59:05.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2270978212 0 0) 0x7f9bf4134020 con 0x7f9bf40a9d70 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf4133e90 con 0x7f9bf40a9d70 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9be40017c0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 10 monclient(hunting): dump: 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:05.265 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 493000590 0 0) 0x7f9be0004eb0 con 0x7f9bf413f650 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9bf4134020 con 0x7f9bf413f650 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9be8001960 con 0x7f9bf413f650 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 10 monclient(hunting): dump: 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:05.266 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 767613847 0 0) 0x7f9bf4133e90 con 0x7f9bf40a9d70 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 >> v1:172.21.15.22:6790/0 conn(0x7f9bf413f650 legacy=0x7f9bf4134370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.269+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 >> v1:172.21.15.22:6789/0 conn(0x7f9bf41457f0 legacy=0x7f9bf4130c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 1 monclient: found mon.b 2024-03-29T10:59:05.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf41dac00 con 0x7f9bf40a9d70 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 10 monclient: _finish_auth 0 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 5 monclient: authenticate success, global_id 4375 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 10 monclient: _renew_subs 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf41dae50 con 0x7f9bf40a9d70 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 10 monclient: _renew_subs 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf4134c60 con 0x7f9bf40a9d70 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.270+0000 7f9bf9b54640 1 librados: init done 2024-03-29T10:59:05.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.271+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9be4001cd0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.271+0000 7f9bdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.271+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9be40035f0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.271+0000 7f9bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.271+0000 7f9bdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.271+0000 7f9bdbfff640 10 monclient: dump: 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:05.269 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.271+0000 7f9bf1ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.272+0000 7f9bf9b54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.272+0000 7f9bf9b54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.272+0000 7f9bf9b54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.272+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd4005080 con 0x7f9bf40a9d70 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.273+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f9be40195c0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.274+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f9be40835a0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.274+0000 7f9bdbfff640 10 client.4375.objecter ms_dispatch 0x7f9bf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.274+0000 7f9bdbfff640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.274+0000 7f9bdbfff640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.274+0000 7f9bdbfff640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.274+0000 7f9bf2ffd640 10 monclient: get_auth_request con 0x7f9be00766d0 auth_method 0 2024-03-29T10:59:05.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.274+0000 7f9bdbfff640 10 client.4375.objecter ms_handle_connect 0x7f9be00766d0 2024-03-29T10:59:05.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.281+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9be404ee80 con 0x7f9bf40a9d70 2024-03-29T10:59:05.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.281+0000 7f9bdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:05.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.281+0000 7f9bdbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:05.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.595+0000 7f9bf9b54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:05.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.595+0000 7f9bf9b54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:05.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.595+0000 7f9bf9b54640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:05.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.595+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bd4005330 con 0x7f9bf40a9d70 2024-03-29T10:59:05.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.596+0000 7f9bdbfff640 1 -- 172.21.15.22:0/2200692727 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f9be4050db0 con 0x7f9bf40a9d70 2024-03-29T10:59:05.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.596+0000 7f9bdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:05.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.596+0000 7f9bdbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:05.576 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:05.577 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:05.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.599+0000 7f9bf9b54640 10 librados: watch_flush enter 2024-03-29T10:59:05.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 10 librados: watch_flush exit 2024-03-29T10:59:05.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-29T10:59:05.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 10 client.4375.objecter successfully canceled tick 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 >> v1:172.21.15.22:6816/380221644 conn(0x7f9be00766d0 legacy=0x7f9be0078b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 10 monclient: shutdown 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 >> v1:172.21.15.26:6789/0 conn(0x7f9bf40a9d70 legacy=0x7f9bf41da4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 shutdown_connections 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 >> 172.21.15.22:0/2200692727 conn(0x7f9bf4066a00 msgr2=0x7f9bf40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 shutdown_connections 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 1 -- 172.21.15.22:0/2200692727 wait complete. 2024-03-29T10:59:05.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:05.600+0000 7f9bf9b54640 1 librados: shutdown 2024-03-29T10:59:05.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:12.405 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:12.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.571+0000 7f30b5a4e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:12.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.571+0000 7f30b5a4e640 10 monclient: build_initial_monmap 2024-03-29T10:59:12.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.571+0000 7f30b5a4e640 10 monclient: monmap: 2024-03-29T10:59:12.551 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:12.551 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:12.551 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:12.572092+0000 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:12.572092+0000 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:12.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.572+0000 7f30b5a4e640 1 Processor -- start 2024-03-29T10:59:12.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.572+0000 7f30b5a4e640 1 -- start start 2024-03-29T10:59:12.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.572+0000 7f30b5a4e640 10 monclient: init 2024-03-29T10:59:12.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:12.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:12.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f30b01457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:12.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f30b013f650 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:12.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f30b00a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:12.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00aaef0 con 0x7f30b01457f0 2024-03-29T10:59:12.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b0145bd0 con 0x7f30b00a9d70 2024-03-29T10:59:12.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b0145eb0 con 0x7f30b013f650 2024-03-29T10:59:12.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:12.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30b5a4e640 10 monclient(hunting): authenticate will time out at 1053.877395s 2024-03-29T10:59:12.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30ae7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f30b013f650 0x7f30b00b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40226/0 (socket says 172.21.15.22:40226) 2024-03-29T10:59:12.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.573+0000 7f30ae7fc640 1 -- 172.21.15.22:0/982748375 learned_addr learned my addr 172.21.15.22:0/982748375 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926788015 0 0) 0x7f30b0145eb0 con 0x7f30b013f650 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): my global_id is 4463 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f309c003520 con 0x7f30b013f650 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38352524 0 0) 0x7f30b00aaef0 con 0x7f30b01457f0 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:12.558 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:12.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 10 monclient(hunting): my global_id is 4383 2024-03-29T10:59:12.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.577+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b0145eb0 con 0x7f30b01457f0 2024-03-29T10:59:12.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.578+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 393836480 0 0) 0x7f309c003520 con 0x7f30b013f650 2024-03-29T10:59:12.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.578+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 >> v1:172.21.15.26:6789/0 conn(0x7f30b00a9d70 legacy=0x7f30b014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:12.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 >> v1:172.21.15.22:6789/0 conn(0x7f30b01457f0 legacy=0x7f30b013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:12.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 1 monclient: found mon.noname-b 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b0146340 con 0x7f30b013f650 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f30a0001690 con 0x7f30b013f650 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.579+0000 7f30adffb640 10 monclient: dump: 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.564 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.580+0000 7f30b5a4e640 5 monclient: authenticate success, global_id 4463 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.580+0000 7f30b5a4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a00017f0 con 0x7f30b013f650 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30b5a4e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30b5a4e640 4 set_mon_vals no callback set 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30adffb640 1 -- 172.21.15.22:0/982748375 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f30a0001c40 con 0x7f30b013f650 2024-03-29T10:59:12.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.583+0000 7f30adffb640 10 monclient: dump: 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:12.566 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.584+0000 7f30b5a4e640 10 monclient: shutdown 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.584+0000 7f30b5a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.584+0000 7f30b5a4e640 1 -- 172.21.15.22:0/982748375 >> v1:172.21.15.22:6790/0 conn(0x7f30b013f650 legacy=0x7f30b00b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.584+0000 7f30b5a4e640 1 -- 172.21.15.22:0/982748375 shutdown_connections 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.584+0000 7f30b5a4e640 1 -- 172.21.15.22:0/982748375 >> 172.21.15.22:0/982748375 conn(0x7f30b0066a00 msgr2=0x7f30b0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.585+0000 7f30b5a4e640 1 -- 172.21.15.22:0/982748375 shutdown_connections 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.585+0000 7f30b5a4e640 1 -- 172.21.15.22:0/982748375 wait complete. 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.586+0000 7f30b5a4e640 10 monclient: build_initial_monmap 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.586+0000 7f30b5a4e640 10 monclient: monmap: 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:12.568 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.587+0000 7f30b5a4e640 1 librados: starting msgr at 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.587+0000 7f30b5a4e640 1 librados: starting objecter 2024-03-29T10:59:12.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.587+0000 7f30b5a4e640 1 Processor -- start 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.587+0000 7f30b5a4e640 1 -- start start 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.587+0000 7f30b5a4e640 1 librados: setting wanted keys 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.587+0000 7f30b5a4e640 1 librados: calling monclient init 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.587+0000 7f30b5a4e640 10 monclient: init 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 10 monclient(hunting): picked mon.noname-c con 0x7f30b00a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 10 monclient(hunting): picked mon.noname-a con 0x7f30b013f650 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 10 monclient(hunting): picked mon.noname-b con 0x7f30b01457f0 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b0142960 con 0x7f30b013f650 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b0142df0 con 0x7f30b01457f0 2024-03-29T10:59:12.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b01e8410 con 0x7f30b00a9d70 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.588+0000 7f30b5a4e640 10 monclient(hunting): authenticate will time out at 1053.892669s 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.589+0000 7f30aeffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f30b00a9d70 0x7f30b006f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40504/0 (socket says 172.21.15.22:40504) 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.589+0000 7f30aeffd640 1 -- 172.21.15.22:0/400091393 learned_addr learned my addr 172.21.15.22:0/400091393 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.589+0000 7f30a7fff640 10 client.?.objecter ms_handle_connect 0x7f30b00a9d70 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.589+0000 7f30a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.590+0000 7f30aeffd640 1 -- 172.21.15.22:0/400091393 >> v1:172.21.15.22:6790/0 conn(0x7f30b00a9d70 legacy=0x7f30b006f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.590+0000 7f30aeffd640 1 -- 172.21.15.22:0/400091393 >> v1:172.21.15.22:6790/0 conn(0x7f30b00a9d70 legacy=0x7f30b006f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.590+0000 7f30aeffd640 1 --1- 172.21.15.22:0/400091393 >> v1:172.21.15.22:6790/0 conn(0x7f30b00a9d70 0x7f30b006f780 :-1 s=OPENED pgs=506992 cs=1 l=1).handle_message read tag failed 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.590+0000 7f30aeffd640 1 --1- 172.21.15.22:0/400091393 >> v1:172.21.15.22:6790/0 conn(0x7f30b00a9d70 0x7f30b006f780 :-1 s=OPENED pgs=506992 cs=1 l=1).fault on lossy channel, failing 2024-03-29T10:59:12.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.590+0000 7f30a7fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.22:6790/0 2024-03-29T10:59:12.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 client.?.objecter ms_handle_connect 0x7f30b013f650 2024-03-29T10:59:12.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:12.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083206291 0 0) 0x7f30b0142960 con 0x7f30b013f650 2024-03-29T10:59:12.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:12.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): my global_id is 4386 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3090004eb0 con 0x7f30b013f650 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2771773628 0 0) 0x7f3090004eb0 con 0x7f30b013f650 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30b0142960 con 0x7f30b013f650 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f30a0001b00 con 0x7f30b013f650 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): dump: 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.577 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 client.?.objecter ms_handle_connect 0x7f30b01457f0 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.593+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3649391878 0 0) 0x7f30b0142960 con 0x7f30b013f650 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 >> v1:172.21.15.22:6790/0 conn(0x7f30b00a9d70 legacy=0x7f30b006f780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 >> v1:172.21.15.26:6789/0 conn(0x7f30b01457f0 legacy=0x7f30b01e7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:12.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 1 monclient: found mon.a 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b013b310 con 0x7f30b013f650 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 5 monclient: authenticate success, global_id 4386 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 10 monclient: _renew_subs 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b013b4a0 con 0x7f30b013f650 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 10 monclient: _renew_subs 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:12.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b0142490 con 0x7f30b013f650 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30b5a4e640 1 librados: init done 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30a0001ca0 con 0x7f30b013f650 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30acff9640 4 set_mon_vals no callback set 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f30a0004d90 con 0x7f30b013f650 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.594+0000 7f30a7fff640 10 monclient: dump: 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:12.580 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:12.581 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:12.581 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:12.581 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:12.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.596+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f30a001ad60 con 0x7f30b013f650 2024-03-29T10:59:12.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30b5a4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:12.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30b5a4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:12.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f30a0084b30 con 0x7f30b013f650 2024-03-29T10:59:12.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30a7fff640 10 client.4386.objecter ms_dispatch 0x7f30b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:12.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30a7fff640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:12.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30a7fff640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:12.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30a7fff640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-29T10:59:12.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30b5a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:12.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.597+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3094005080 con 0x7f30b013f650 2024-03-29T10:59:12.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.599+0000 7f30aeffd640 10 monclient: get_auth_request con 0x7f309006dc50 auth_method 0 2024-03-29T10:59:12.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.599+0000 7f30a7fff640 10 client.4386.objecter ms_handle_connect 0x7f309006dc50 2024-03-29T10:59:12.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.606+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30a0050240 con 0x7f30b013f650 2024-03-29T10:59:12.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.606+0000 7f30a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:12.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.606+0000 7f30a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:12.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.933+0000 7f30b5a4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:12.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.933+0000 7f30b5a4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:12.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.933+0000 7f30b5a4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:12.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.933+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3094005330 con 0x7f30b013f650 2024-03-29T10:59:12.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.936+0000 7f30a7fff640 1 -- 172.21.15.22:0/400091393 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f30a0052340 con 0x7f30b013f650 2024-03-29T10:59:12.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.936+0000 7f30a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:12.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.937+0000 7f30a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:12.920 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:12.920 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:12.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 10 librados: watch_flush enter 2024-03-29T10:59:12.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 10 librados: watch_flush exit 2024-03-29T10:59:12.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-29T10:59:12.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 10 client.4386.objecter successfully canceled tick 2024-03-29T10:59:12.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 >> v1:172.21.15.22:6816/380221644 conn(0x7f309006dc50 legacy=0x7f3090070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:12.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 10 monclient: shutdown 2024-03-29T10:59:12.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:12.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.940+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 >> v1:172.21.15.22:6789/0 conn(0x7f30b013f650 legacy=0x7f30b01e75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:12.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.942+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 shutdown_connections 2024-03-29T10:59:12.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.942+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 >> 172.21.15.22:0/400091393 conn(0x7f30b0066a00 msgr2=0x7f30b00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:12.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.942+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 shutdown_connections 2024-03-29T10:59:12.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.942+0000 7f30b5a4e640 1 -- 172.21.15.22:0/400091393 wait complete. 2024-03-29T10:59:12.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:12.942+0000 7f30b5a4e640 1 librados: shutdown 2024-03-29T10:59:12.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:19.747 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.916+0000 7f8401160640 10 monclient: get_monmap_and_config 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.916+0000 7f8401160640 10 monclient: build_initial_monmap 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.917+0000 7f8401160640 10 monclient: monmap: 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:19.917784+0000 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:19.917784+0000 2024-03-29T10:59:19.897 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.918+0000 7f8401160640 1 Processor -- start 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.918+0000 7f8401160640 1 -- start start 2024-03-29T10:59:19.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.918+0000 7f8401160640 10 monclient: init 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 10 monclient(hunting): picked mon.noname-a con 0x7f83fc0b1f60 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 10 monclient(hunting): picked mon.noname-b con 0x7f83fc071eb0 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 10 monclient(hunting): picked mon.noname-c con 0x7f83fc143d10 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83fc06ef80 con 0x7f83fc0b1f60 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83fc06f260 con 0x7f83fc143d10 2024-03-29T10:59:19.903 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83fc0b40c0 con 0x7f83fc071eb0 2024-03-29T10:59:19.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:19.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.919+0000 7f8401160640 10 monclient(hunting): authenticate will time out at 1061.223918s 2024-03-29T10:59:19.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.922+0000 7f83fad9d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f83fc0b1f60 0x7f83fc143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:32891/0 (socket says 172.21.15.22:32891) 2024-03-29T10:59:19.904 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.922+0000 7f83fad9d640 1 -- 172.21.15.22:0/3108259641 learned_addr learned my addr 172.21.15.22:0/3108259641 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:19.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2379012772 0 0) 0x7f83fc06ef80 con 0x7f83fc0b1f60 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): my global_id is 4389 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e4003520 con 0x7f83fc0b1f60 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442816755 0 0) 0x7f83fc0b40c0 con 0x7f83fc071eb0 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): _negotiate 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 10 monclient(hunting): my global_id is 4469 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83fc06ef80 con 0x7f83fc071eb0 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201794716 0 0) 0x7f83e4003520 con 0x7f83fc0b1f60 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 >> v1:172.21.15.22:6790/0 conn(0x7f83fc071eb0 legacy=0x7f83fc146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.924+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 >> v1:172.21.15.26:6789/0 conn(0x7f83fc143d10 legacy=0x7f83fc146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:19.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: _finish_hunting 0 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 1 monclient: found mon.noname-a 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc155d70 con 0x7f83fc0b1f60 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: _finish_auth 0 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: _check_auth_tickets 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f8401160640 5 monclient: authenticate success, global_id 4389 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f8401160640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f83f4001a10 con 0x7f83fc0b1f60 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: dump: 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.909 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f8401160640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83f4001c30 con 0x7f83fc0b1f60 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f8401160640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f8401160640 4 set_mon_vals no callback set 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 1 -- 172.21.15.22:0/3108259641 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f83f4001fc0 con 0x7f83fc0b1f60 2024-03-29T10:59:19.910 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:19.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:19.911 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.925+0000 7f83f9d9b640 10 monclient: dump: 2024-03-29T10:59:19.911 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:19.911 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.926+0000 7f8401160640 10 monclient: shutdown 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.926+0000 7f8401160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.926+0000 7f8401160640 1 -- 172.21.15.22:0/3108259641 >> v1:172.21.15.22:6789/0 conn(0x7f83fc0b1f60 legacy=0x7f83fc143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.926+0000 7f8401160640 1 -- 172.21.15.22:0/3108259641 shutdown_connections 2024-03-29T10:59:19.912 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.926+0000 7f8401160640 1 -- 172.21.15.22:0/3108259641 >> 172.21.15.22:0/3108259641 conn(0x7f83fc066a00 msgr2=0x7f83fc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.929+0000 7f8401160640 1 -- 172.21.15.22:0/3108259641 shutdown_connections 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.929+0000 7f8401160640 1 -- 172.21.15.22:0/3108259641 wait complete. 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.930+0000 7f8401160640 10 monclient: build_initial_monmap 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 10 monclient: monmap: 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:19.914 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 1 librados: starting msgr at 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 1 librados: starting objecter 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 1 Processor -- start 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 1 -- start start 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 1 librados: setting wanted keys 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 1 librados: calling monclient init 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.931+0000 7f8401160640 10 monclient: init 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:19.915 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 10 monclient(hunting): picked mon.noname-b con 0x7f83fc0b1f60 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 10 monclient(hunting): picked mon.noname-a con 0x7f83fc143d10 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 10 monclient(hunting): picked mon.noname-c con 0x7f83fc145b10 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83fc149630 con 0x7f83fc143d10 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83fc14ae60 con 0x7f83fc0b1f60 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83fc14b140 con 0x7f83fc145b10 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:19.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.932+0000 7f8401160640 10 monclient(hunting): authenticate will time out at 1061.236841s 2024-03-29T10:59:19.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83fad9d640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f83fc0b1f60 0x7f83fc149da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:60729/0 (socket says 172.21.15.22:60729) 2024-03-29T10:59:19.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83fad9d640 1 -- 172.21.15.22:0/833933723 learned_addr learned my addr 172.21.15.22:0/833933723 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:19.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 10 client.?.objecter ms_handle_connect 0x7f83fc0b1f60 2024-03-29T10:59:19.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:19.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 458619022 0 0) 0x7f83fc14ae60 con 0x7f83fc0b1f60 2024-03-29T10:59:19.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:19.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:19.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:19.918 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 10 monclient(hunting): my global_id is 4381 2024-03-29T10:59:19.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.935+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e8004eb0 con 0x7f83fc0b1f60 2024-03-29T10:59:19.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 client.?.objecter ms_handle_connect 0x7f83fc145b10 2024-03-29T10:59:19.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1907993949 0 0) 0x7f83e8004eb0 con 0x7f83fc0b1f60 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83fc14ae60 con 0x7f83fc0b1f60 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 client.?.objecter ms_handle_connect 0x7f83fc143d10 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f83f4001a50 con 0x7f83fc0b1f60 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): dump: 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:19.920 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860209947 0 0) 0x7f83fc14b140 con 0x7f83fc145b10 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): my global_id is 4472 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e8004eb0 con 0x7f83fc145b10 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257152381 0 0) 0x7f83fc149630 con 0x7f83fc143d10 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:19.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 10 monclient(hunting): my global_id is 4392 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.938+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83fc14b140 con 0x7f83fc143d10 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 754587642 0 0) 0x7f83fc14b140 con 0x7f83fc143d10 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83fc149630 con 0x7f83fc143d10 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f83f00018f0 con 0x7f83fc143d10 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 10 monclient(hunting): dump: 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:19.922 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1002494820 0 0) 0x7f83fc14ae60 con 0x7f83fc0b1f60 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 >> v1:172.21.15.22:6790/0 conn(0x7f83fc145b10 legacy=0x7f83fc1e5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 >> v1:172.21.15.22:6789/0 conn(0x7f83fc143d10 legacy=0x7f83fc147320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 monclient: found mon.b 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83fc14b420 con 0x7f83fc0b1f60 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f83e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:19.923 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 5 monclient: authenticate success, global_id 4381 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 10 monclient: _renew_subs 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83fc149030 con 0x7f83fc0b1f60 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 10 monclient: _renew_subs 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83fc1e75c0 con 0x7f83fc0b1f60 2024-03-29T10:59:19.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.939+0000 7f8401160640 1 librados: init done 2024-03-29T10:59:19.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.941+0000 7f8401160640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:19.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.941+0000 7f8401160640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:19.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.941+0000 7f8401160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:19.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.941+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83fc075f10 con 0x7f83fc0b1f60 2024-03-29T10:59:19.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.941+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83f4004e10 con 0x7f83fc0b1f60 2024-03-29T10:59:19.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.941+0000 7f83e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:19.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.942+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f83f40051e0 con 0x7f83fc0b1f60 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.942+0000 7f83e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.942+0000 7f83e37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.942+0000 7f83e37fe640 10 monclient: dump: 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.941+0000 7f83f959a640 4 set_mon_vals no callback set 2024-03-29T10:59:19.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.943+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f83f4005400 con 0x7f83fc0b1f60 2024-03-29T10:59:19.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.944+0000 7f83fa59c640 10 monclient: get_auth_request con 0x7f83e806e000 auth_method 0 2024-03-29T10:59:19.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.944+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f83f4084fa0 con 0x7f83fc0b1f60 2024-03-29T10:59:19.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.944+0000 7f83e37fe640 10 client.4381.objecter ms_dispatch 0x7f83fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:19.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.944+0000 7f83e37fe640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:19.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.944+0000 7f83e37fe640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:19.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.944+0000 7f83e37fe640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-29T10:59:19.928 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.945+0000 7f83e37fe640 10 client.4381.objecter ms_handle_connect 0x7f83e806e000 2024-03-29T10:59:19.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.951+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83f4050920 con 0x7f83fc0b1f60 2024-03-29T10:59:19.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.951+0000 7f83e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:19.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:19.951+0000 7f83e37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:20.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.281+0000 7f8401160640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:20.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.281+0000 7f8401160640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:20.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.281+0000 7f8401160640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T10:59:20.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.281+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83fc149980 con 0x7f83fc0b1f60 2024-03-29T10:59:20.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.282+0000 7f83e37fe640 1 -- 172.21.15.22:0/833933723 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f83f4052850 con 0x7f83fc0b1f60 2024-03-29T10:59:20.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.282+0000 7f83e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:20.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.282+0000 7f83e37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:20.262 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:20.262 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:20.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.285+0000 7f8401160640 10 librados: watch_flush enter 2024-03-29T10:59:20.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.285+0000 7f8401160640 10 librados: watch_flush exit 2024-03-29T10:59:20.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.285+0000 7f8401160640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-29T10:59:20.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.285+0000 7f8401160640 10 client.4381.objecter successfully canceled tick 2024-03-29T10:59:20.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.285+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 >> v1:172.21.15.22:6816/380221644 conn(0x7f83e806e000 legacy=0x7f83e80704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:20.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 10 monclient: shutdown 2024-03-29T10:59:20.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:20.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 >> v1:172.21.15.26:6789/0 conn(0x7f83fc0b1f60 legacy=0x7f83fc149da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:20.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 shutdown_connections 2024-03-29T10:59:20.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 >> 172.21.15.22:0/833933723 conn(0x7f83fc066a00 msgr2=0x7f83fc1428f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:20.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 shutdown_connections 2024-03-29T10:59:20.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 1 -- 172.21.15.22:0/833933723 wait complete. 2024-03-29T10:59:20.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:20.286+0000 7f8401160640 1 librados: shutdown 2024-03-29T10:59:20.287 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:27.094 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.265+0000 7f2d31c0e640 10 monclient: get_monmap_and_config 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.265+0000 7f2d31c0e640 10 monclient: build_initial_monmap 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.265+0000 7f2d31c0e640 10 monclient: monmap: 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:27.265941+0000 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:27.265941+0000 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:27.250 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.266+0000 7f2d31c0e640 1 Processor -- start 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.266+0000 7f2d31c0e640 1 -- start start 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.266+0000 7f2d31c0e640 10 monclient: init 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.266+0000 7f2d31c0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.266+0000 7f2d31c0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.266+0000 7f2d31c0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d2c0ad6c0 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d31c0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d2c0af6c0 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d31c0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2c14fdc0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d31c0e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c0a94e0 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.251 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d31c0e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c0af3a0 con 0x7f2d2c0af6c0 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d31c0e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c0afb60 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d31c0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d31c0e640 10 monclient(hunting): authenticate will time out at 1068.571161s 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f2d2c0ad6c0 0x7f2d2c14f6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56522/0 (socket says 172.21.15.22:56522) 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2b7fe640 1 -- 172.21.15.22:0/1659060132 learned_addr learned my addr 172.21.15.22:0/1659060132 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2421730807 0 0) 0x7f2d2c0afb60 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 10 monclient(hunting): _negotiate 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 10 monclient(hunting): my global_id is 4475 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d14003520 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2341122511 0 0) 0x7f2d14003520 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.252 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.267+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 >> v1:172.21.15.26:6789/0 conn(0x7f2d2c0af6c0 legacy=0x7f2d2c152980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:27.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 >> v1:172.21.15.22:6789/0 conn(0x7f2d2c14fdc0 legacy=0x7f2d2c15a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T10:59:27.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 10 monclient: _finish_hunting 0 2024-03-29T10:59:27.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 1 monclient: found mon.noname-b 2024-03-29T10:59:27.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c0aab80 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 10 monclient: _finish_auth 0 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 10 monclient: _check_auth_tickets 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2d1c001690 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d2a7fc640 10 monclient: dump: 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.254 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d31c0e640 5 monclient: authenticate success, global_id 4475 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.270+0000 7f2d31c0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d1c001860 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d2a7fc640 1 -- 172.21.15.22:0/1659060132 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2d1c001cb0 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d2a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d2a7fc640 10 monclient: dump: 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:27.255 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d31c0e640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d31c0e640 4 set_mon_vals no callback set 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d31c0e640 10 monclient: shutdown 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d31c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.271+0000 7f2d31c0e640 1 -- 172.21.15.22:0/1659060132 >> v1:172.21.15.22:6790/0 conn(0x7f2d2c0ad6c0 legacy=0x7f2d2c14f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.272+0000 7f2d31c0e640 1 -- 172.21.15.22:0/1659060132 shutdown_connections 2024-03-29T10:59:27.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.272+0000 7f2d31c0e640 1 -- 172.21.15.22:0/1659060132 >> 172.21.15.22:0/1659060132 conn(0x7f2d2c0b35e0 msgr2=0x7f2d2c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:27.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.272+0000 7f2d31c0e640 1 -- 172.21.15.22:0/1659060132 shutdown_connections 2024-03-29T10:59:27.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.273+0000 7f2d31c0e640 1 -- 172.21.15.22:0/1659060132 wait complete. 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.274+0000 7f2d31c0e640 10 monclient: build_initial_monmap 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.274+0000 7f2d31c0e640 10 monclient: monmap: 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:27.258 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.274+0000 7f2d31c0e640 1 librados: starting msgr at 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.274+0000 7f2d31c0e640 1 librados: starting objecter 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.275+0000 7f2d31c0e640 1 Processor -- start 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.275+0000 7f2d31c0e640 1 -- start start 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.275+0000 7f2d31c0e640 1 librados: setting wanted keys 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.275+0000 7f2d31c0e640 1 librados: calling monclient init 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.275+0000 7f2d31c0e640 10 monclient: init 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.276+0000 7f2d31c0e640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:27.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.276+0000 7f2d31c0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:27.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.278+0000 7f2d2b7fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f2d2c0ad6c0 0x7f2d2c1e72c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:46005/0 (socket says 172.21.15.22:46005) 2024-03-29T10:59:27.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 10 monclient(hunting): picked mon.noname-a con 0x7f2d2c0ad6c0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d2b7fe640 1 -- 172.21.15.22:0/53131181 learned_addr learned my addr 172.21.15.22:0/53131181 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d13fff640 10 client.?.objecter ms_handle_connect 0x7f2d2c0ad6c0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 10 monclient(hunting): picked mon.noname-b con 0x7f2d2c0af6c0 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 10 monclient(hunting): picked mon.noname-c con 0x7f2d2c14fdc0 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c1e7d40 con 0x7f2d2c0ad6c0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c1e81d0 con 0x7f2d2c0af6c0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c072460 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.279+0000 7f2d31c0e640 10 monclient(hunting): authenticate will time out at 1068.583714s 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 10 client.?.objecter ms_handle_connect 0x7f2d2c14fdc0 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:27.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2013145208 0 0) 0x7f2d2c072460 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 10 monclient(hunting): my global_id is 4478 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.280+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d18004eb0 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2869866216 0 0) 0x7f2d18004eb0 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d2c072460 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2d24001ad0 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 10 monclient(hunting): dump: 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.263 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1624324807 0 0) 0x7f2d2c072460 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.281+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 >> v1:172.21.15.26:6789/0 conn(0x7f2d2c0af6c0 legacy=0x7f2d2c1e3d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:27.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 >> v1:172.21.15.22:6789/0 conn(0x7f2d2c0ad6c0 legacy=0x7f2d2c1e72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:27.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:27.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 1 monclient: found mon.c 2024-03-29T10:59:27.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:27.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:27.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c1e79d0 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 5 monclient: authenticate success, global_id 4478 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 10 monclient: _renew_subs 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c0aedc0 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 10 monclient: _renew_subs 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.282+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c1e4470 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.283+0000 7f2d31c0e640 1 librados: init done 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.283+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d24001c30 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.283+0000 7f2d13fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:27.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.283+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2d24003b10 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.283+0000 7f2d13fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.283+0000 7f2d13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.283+0000 7f2d13fff640 10 monclient: dump: 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:27.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.284+0000 7f2d29ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:27.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.285+0000 7f2d31c0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:27.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.286+0000 7f2d31c0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:27.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.286+0000 7f2d31c0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:27.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.286+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0c005080 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.269 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.286+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f2d24019ae0 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.287+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f2d24083b10 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.287+0000 7f2d13fff640 10 client.4478.objecter ms_dispatch 0x7f2d2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:27.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.287+0000 7f2d2b7fe640 10 monclient: get_auth_request con 0x7f2d1806dc60 auth_method 0 2024-03-29T10:59:27.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.287+0000 7f2d13fff640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:27.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.287+0000 7f2d13fff640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:27.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.287+0000 7f2d13fff640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T10:59:27.270 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.287+0000 7f2d13fff640 10 client.4478.objecter ms_handle_connect 0x7f2d1806dc60 2024-03-29T10:59:27.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.296+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2d2404f220 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.296+0000 7f2d13fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:27.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.296+0000 7f2d13fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:27.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.626+0000 7f2d31c0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:27.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.626+0000 7f2d31c0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:27.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.626+0000 7f2d31c0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:27.606 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.626+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2d0c005330 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.629+0000 7f2d13fff640 1 -- 172.21.15.22:0/53131181 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f2d24051320 con 0x7f2d2c14fdc0 2024-03-29T10:59:27.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.630+0000 7f2d13fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:27.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.630+0000 7f2d13fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:27.611 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:27.611 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:27.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 10 librados: watch_flush enter 2024-03-29T10:59:27.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 10 librados: watch_flush exit 2024-03-29T10:59:27.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T10:59:27.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 10 client.4478.objecter successfully canceled tick 2024-03-29T10:59:27.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 >> v1:172.21.15.22:6816/380221644 conn(0x7f2d1806dc60 legacy=0x7f2d18070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:27.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 10 monclient: shutdown 2024-03-29T10:59:27.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:27.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.635+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 >> v1:172.21.15.22:6790/0 conn(0x7f2d2c14fdc0 legacy=0x7f2d2c1ee860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:27.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.636+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 shutdown_connections 2024-03-29T10:59:27.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.636+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 >> 172.21.15.22:0/53131181 conn(0x7f2d2c0b35e0 msgr2=0x7f2d2c0a94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:27.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.638+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 shutdown_connections 2024-03-29T10:59:27.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.638+0000 7f2d31c0e640 1 -- 172.21.15.22:0/53131181 wait complete. 2024-03-29T10:59:27.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:27.638+0000 7f2d31c0e640 1 librados: shutdown 2024-03-29T10:59:27.635 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:34.441 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:34.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.610+0000 7f0ea25fb640 10 monclient: get_monmap_and_config 2024-03-29T10:59:34.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.610+0000 7f0ea25fb640 10 monclient: build_initial_monmap 2024-03-29T10:59:34.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.610+0000 7f0ea25fb640 10 monclient: monmap: 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:34.611410+0000 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:34.611410+0000 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:34.591 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:34.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.611+0000 7f0ea25fb640 1 Processor -- start 2024-03-29T10:59:34.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.611+0000 7f0ea25fb640 1 -- start start 2024-03-29T10:59:34.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.611+0000 7f0ea25fb640 10 monclient: init 2024-03-29T10:59:34.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:34.593 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c13f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c0aaef0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c145bd0 con 0x7f0e9c13f650 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c145eb0 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:34.594 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.612+0000 7f0ea25fb640 10 monclient(hunting): authenticate will time out at 1075.916739s 2024-03-29T10:59:34.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.615+0000 7f0ea0b98640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0e9c0a9d70 0x7f0e9c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59412/0 (socket says 172.21.15.22:59412) 2024-03-29T10:59:34.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.615+0000 7f0ea0b98640 1 -- 172.21.15.22:0/1379953839 learned_addr learned my addr 172.21.15.22:0/1379953839 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:34.596 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832313535 0 0) 0x7f0e9c145eb0 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 10 monclient(hunting): _negotiate 2024-03-29T10:59:34.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:34.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:34.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 10 monclient(hunting): my global_id is 4481 2024-03-29T10:59:34.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e84003520 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 798531208 0 0) 0x7f0e84003520 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.597 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.616+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 >> v1:172.21.15.26:6789/0 conn(0x7f0e9c13f650 legacy=0x7f0e9c0b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:34.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 >> v1:172.21.15.22:6789/0 conn(0x7f0e9c1457f0 legacy=0x7f0e9c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:34.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: _finish_hunting 0 2024-03-29T10:59:34.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 1 monclient: found mon.noname-b 2024-03-29T10:59:34.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:34.600 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c146340 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: _finish_auth 0 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: _check_auth_tickets 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0e94001a10 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: dump: 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:34.601 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0ea25fb640 5 monclient: authenticate success, global_id 4481 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0ea25fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e94001be0 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 1 -- 172.21.15.22:0/1379953839 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0e94001ff0 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.617+0000 7f0e9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.618+0000 7f0e9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.618+0000 7f0e9affd640 10 monclient: dump: 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:34.602 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.618+0000 7f0ea25fb640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.618+0000 7f0ea25fb640 4 set_mon_vals no callback set 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.618+0000 7f0ea25fb640 10 monclient: shutdown 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.618+0000 7f0ea25fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.618+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1379953839 >> v1:172.21.15.22:6790/0 conn(0x7f0e9c0a9d70 legacy=0x7f0e9c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.619+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1379953839 shutdown_connections 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.619+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1379953839 >> 172.21.15.22:0/1379953839 conn(0x7f0e9c066a00 msgr2=0x7f0e9c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:34.603 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.619+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1379953839 shutdown_connections 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.619+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1379953839 wait complete. 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.620+0000 7f0ea25fb640 10 monclient: build_initial_monmap 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.620+0000 7f0ea25fb640 10 monclient: monmap: 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:34.604 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.621+0000 7f0ea25fb640 1 librados: starting msgr at 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.621+0000 7f0ea25fb640 1 librados: starting objecter 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.621+0000 7f0ea25fb640 1 Processor -- start 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.621+0000 7f0ea25fb640 1 -- start start 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.621+0000 7f0ea25fb640 1 librados: setting wanted keys 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.621+0000 7f0ea25fb640 1 librados: calling monclient init 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.621+0000 7f0ea25fb640 10 monclient: init 2024-03-29T10:59:34.609 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 10 monclient(hunting): picked mon.noname-c con 0x7f0e9c0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 10 monclient(hunting): picked mon.noname-b con 0x7f0e9c13f650 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 10 monclient(hunting): picked mon.noname-a con 0x7f0e9c1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c141980 con 0x7f0e9c1457f0 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c13fda0 con 0x7f0e9c13f650 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e9c141e10 con 0x7f0e9c0a9d70 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.622+0000 7f0ea25fb640 10 monclient(hunting): authenticate will time out at 1075.926456s 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.624+0000 7f0e9bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0e9c0a9d70 0x7f0e9c140b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59900/0 (socket says 172.21.15.22:59900) 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.624+0000 7f0e9bfff640 1 -- 172.21.15.22:0/1332851555 learned_addr learned my addr 172.21.15.22:0/1332851555 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 client.?.objecter ms_handle_connect 0x7f0e9c13f650 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:34.610 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1950602621 0 0) 0x7f0e9c13fda0 con 0x7f0e9c13f650 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 monclient(hunting): my global_id is 4384 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e88004eb0 con 0x7f0e9c13f650 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 client.?.objecter ms_handle_connect 0x7f0e9c0a9d70 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.625+0000 7f0e98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 10 client.?.objecter ms_handle_connect 0x7f0e9c1457f0 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930372298 0 0) 0x7f0e9c141980 con 0x7f0e9c1457f0 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 10 monclient(hunting): _negotiate 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 10 monclient(hunting): my global_id is 4401 2024-03-29T10:59:34.611 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.626+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e9c13fda0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972200712 0 0) 0x7f0e88004eb0 con 0x7f0e9c13f650 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e9c141980 con 0x7f0e9c13f650 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4245729351 0 0) 0x7f0e9c13fda0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e88004eb0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0e90001870 con 0x7f0e9c13f650 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:34.612 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0e94001ef0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.627+0000 7f0e98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 10 monclient(hunting): dump: 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:34.613 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 496298012 0 0) 0x7f0e88004eb0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 >> v1:172.21.15.22:6790/0 conn(0x7f0e9c0a9d70 legacy=0x7f0e9c140b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 >> v1:172.21.15.26:6789/0 conn(0x7f0e9c13f650 legacy=0x7f0e9c137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 10 monclient: _finish_hunting 0 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 1 monclient: found mon.a 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e9c142c90 con 0x7f0e9c1457f0 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 10 monclient: _finish_auth 0 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 10 monclient: _check_auth_tickets 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0e98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:34.614 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0ea25fb640 5 monclient: authenticate success, global_id 4401 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0ea25fb640 10 monclient: _renew_subs 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0ea25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.628+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9c141340 con 0x7f0e9c1457f0 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0ea25fb640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0ea25fb640 10 monclient: _renew_subs 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0ea25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9c137740 con 0x7f0e9c1457f0 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0ea25fb640 1 librados: init done 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e94001390 con 0x7f0e9c1457f0 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0e98ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0e94004f70 con 0x7f0e9c1457f0 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0e98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0e98ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0e98ff9640 10 monclient: dump: 2024-03-29T10:59:34.615 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:34.616 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.629+0000 7f0e99ffb640 4 set_mon_vals no callback set 2024-03-29T10:59:34.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.632+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f0e9401af40 con 0x7f0e9c1457f0 2024-03-29T10:59:34.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.633+0000 7f0e9bfff640 10 monclient: get_auth_request con 0x7f0e8806df60 auth_method 0 2024-03-29T10:59:34.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0ea25fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:34.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0ea25fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:34.617 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0ea25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:34.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7c005080 con 0x7f0e9c1457f0 2024-03-29T10:59:34.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0e98ff9640 10 client.4401.objecter ms_handle_connect 0x7f0e8806df60 2024-03-29T10:59:34.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0e94084cc0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0e98ff9640 10 client.4401.objecter ms_dispatch 0x7f0e9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:34.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0e98ff9640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:34.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0e98ff9640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:34.618 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.634+0000 7f0e98ff9640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-29T10:59:34.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.645+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e940505a0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.645+0000 7f0e98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:34.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.645+0000 7f0e98ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:34.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.976+0000 7f0ea25fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:34.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.976+0000 7f0ea25fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:34.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.976+0000 7f0ea25fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T10:59:34.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.976+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e7c005330 con 0x7f0e9c1457f0 2024-03-29T10:59:34.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.977+0000 7f0e98ff9640 1 -- 172.21.15.22:0/1332851555 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0e940524d0 con 0x7f0e9c1457f0 2024-03-29T10:59:34.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.977+0000 7f0e98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:34.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.977+0000 7f0e98ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:34.958 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:34.958 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:34.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.982+0000 7f0ea25fb640 10 librados: watch_flush enter 2024-03-29T10:59:34.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.983+0000 7f0ea25fb640 10 librados: watch_flush exit 2024-03-29T10:59:34.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.983+0000 7f0ea25fb640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-29T10:59:34.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.983+0000 7f0ea25fb640 10 client.4401.objecter successfully canceled tick 2024-03-29T10:59:34.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.983+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 >> v1:172.21.15.22:6816/380221644 conn(0x7f0e8806df60 legacy=0x7f0e88070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:34.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.983+0000 7f0ea25fb640 10 monclient: shutdown 2024-03-29T10:59:34.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.983+0000 7f0ea25fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:34.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.983+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 >> v1:172.21.15.22:6789/0 conn(0x7f0e9c1457f0 legacy=0x7f0e9c071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:34.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.984+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 shutdown_connections 2024-03-29T10:59:34.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.984+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 >> 172.21.15.22:0/1332851555 conn(0x7f0e9c066a00 msgr2=0x7f0e9c0b4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:34.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.984+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 shutdown_connections 2024-03-29T10:59:34.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.984+0000 7f0ea25fb640 1 -- 172.21.15.22:0/1332851555 wait complete. 2024-03-29T10:59:34.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:34.985+0000 7f0ea25fb640 1 librados: shutdown 2024-03-29T10:59:34.982 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:41.788 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.950+0000 7f1a65bad640 10 monclient: get_monmap_and_config 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.950+0000 7f1a65bad640 10 monclient: build_initial_monmap 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.950+0000 7f1a65bad640 10 monclient: monmap: 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:41.951746+0000 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:41.951746+0000 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:41.931 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:41.932 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:41.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.953+0000 7f1a65bad640 1 Processor -- start 2024-03-29T10:59:41.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.953+0000 7f1a65bad640 1 -- start start 2024-03-29T10:59:41.933 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.953+0000 7f1a65bad640 10 monclient: init 2024-03-29T10:59:41.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:41.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 10 monclient(hunting): picked mon.noname-a con 0x7f1a580389e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 10 monclient(hunting): picked mon.noname-b con 0x7f1a58038dc0 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 10 monclient(hunting): picked mon.noname-c con 0x7f1a58030660 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a580c54c0 con 0x7f1a580389e0 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a58036430 con 0x7f1a58030660 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a58039840 con 0x7f1a58038dc0 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:41.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.954+0000 7f1a65bad640 10 monclient(hunting): authenticate will time out at 1083.258368s 2024-03-29T10:59:41.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5ffff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f1a58038dc0 0x7f1a580cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60044/0 (socket says 172.21.15.22:60044) 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5ffff640 1 -- 172.21.15.22:0/1084858282 learned_addr learned my addr 172.21.15.22:0/1084858282 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001996913 0 0) 0x7f1a58039840 con 0x7f1a58038dc0 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5f7fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5f7fe640 10 monclient(hunting): my global_id is 4487 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.956+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a50003520 con 0x7f1a58038dc0 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.957+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2421216163 0 0) 0x7f1a50003520 con 0x7f1a58038dc0 2024-03-29T10:59:41.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.957+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 >> v1:172.21.15.26:6789/0 conn(0x7f1a58030660 legacy=0x7f1a580d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:41.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.959+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 >> v1:172.21.15.22:6789/0 conn(0x7f1a580389e0 legacy=0x7f1a58033520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:41.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:41.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 1 monclient: found mon.noname-b 2024-03-29T10:59:41.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:41.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:59:41.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58035690 con 0x7f1a58038dc0 2024-03-29T10:59:41.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:41.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:41.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:41.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a65bad640 5 monclient: authenticate success, global_id 4487 2024-03-29T10:59:41.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a65bad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1a54001690 con 0x7f1a58038dc0 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a5f7fe640 10 monclient: dump: 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.943 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.960+0000 7f1a65bad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a54001860 con 0x7f1a58038dc0 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a5f7fe640 1 -- 172.21.15.22:0/1084858282 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1a54001cb0 con 0x7f1a58038dc0 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a5f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a5f7fe640 10 monclient: dump: 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:41.944 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:41.945 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:41.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a65bad640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:41.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.963+0000 7f1a65bad640 4 set_mon_vals no callback set 2024-03-29T10:59:41.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.964+0000 7f1a65bad640 10 monclient: shutdown 2024-03-29T10:59:41.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.964+0000 7f1a65bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.964+0000 7f1a65bad640 1 -- 172.21.15.22:0/1084858282 >> v1:172.21.15.22:6790/0 conn(0x7f1a58038dc0 legacy=0x7f1a580cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.964+0000 7f1a65bad640 1 -- 172.21.15.22:0/1084858282 shutdown_connections 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.964+0000 7f1a65bad640 1 -- 172.21.15.22:0/1084858282 >> 172.21.15.22:0/1084858282 conn(0x7f1a58005c00 msgr2=0x7f1a58006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.964+0000 7f1a65bad640 1 -- 172.21.15.22:0/1084858282 shutdown_connections 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.964+0000 7f1a65bad640 1 -- 172.21.15.22:0/1084858282 wait complete. 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.965+0000 7f1a65bad640 10 monclient: build_initial_monmap 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.965+0000 7f1a65bad640 10 monclient: monmap: 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:41.948 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.966+0000 7f1a65bad640 1 librados: starting msgr at 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.966+0000 7f1a65bad640 1 librados: starting objecter 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.966+0000 7f1a65bad640 1 Processor -- start 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.966+0000 7f1a65bad640 1 -- start start 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.966+0000 7f1a65bad640 1 librados: setting wanted keys 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.966+0000 7f1a65bad640 1 librados: calling monclient init 2024-03-29T10:59:41.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.966+0000 7f1a65bad640 10 monclient: init 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 10 monclient(hunting): picked mon.noname-b con 0x7f1a58030660 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 10 monclient(hunting): picked mon.noname-a con 0x7f1a580389e0 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 10 monclient(hunting): picked mon.noname-c con 0x7f1a58160550 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a5802ec50 con 0x7f1a580389e0 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a58160030 con 0x7f1a58030660 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a58030010 con 0x7f1a58160550 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a65bad640 10 monclient(hunting): authenticate will time out at 1083.272217s 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a653ac640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f1a58160550 0x7f1a58167220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60340/0 (socket says 172.21.15.22:60340) 2024-03-29T10:59:41.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.968+0000 7f1a653ac640 1 -- 172.21.15.22:0/52425276 learned_addr learned my addr 172.21.15.22:0/52425276 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:41.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 10 client.?.objecter ms_handle_connect 0x7f1a58160550 2024-03-29T10:59:41.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:41.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557205172 0 0) 0x7f1a58030010 con 0x7f1a58160550 2024-03-29T10:59:41.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 10 monclient(hunting): _negotiate 2024-03-29T10:59:41.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:41.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:41.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 10 monclient(hunting): my global_id is 4490 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.970+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a44004eb0 con 0x7f1a58160550 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.972+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 587627696 0 0) 0x7f1a44004eb0 con 0x7f1a58160550 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.972+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1a58030010 con 0x7f1a58160550 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.972+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1a6004fdd0 con 0x7f1a58160550 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.972+0000 7f1a5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.972+0000 7f1a5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.972+0000 7f1a5d7fa640 10 monclient(hunting): dump: 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:41.954 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:41.955 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:41.955 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:41.955 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:41.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.972+0000 7f1a5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:41.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.973+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103309782 0 0) 0x7f1a58030010 con 0x7f1a58160550 2024-03-29T10:59:41.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.973+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 >> v1:172.21.15.26:6789/0 conn(0x7f1a58030660 legacy=0x7f1a5802d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:41.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.973+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 >> v1:172.21.15.22:6789/0 conn(0x7f1a580389e0 legacy=0x7f1a580c7e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 10 monclient: _finish_hunting 0 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 1 monclient: found mon.c 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a581609c0 con 0x7f1a58160550 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 10 monclient: _finish_auth 0 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 10 monclient: _check_auth_tickets 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 10 client.?.objecter ms_handle_connect 0x7f1a58030660 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:41.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 5 monclient: authenticate success, global_id 4490 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 10 monclient: _renew_subs 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a58160c10 con 0x7f1a58160550 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 10 monclient: _renew_subs 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a58168b20 con 0x7f1a58160550 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.976+0000 7f1a65bad640 1 librados: init done 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.977+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1a6004f400 con 0x7f1a58160550 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.977+0000 7f1a5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.977+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1a60075f10 con 0x7f1a58160550 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.977+0000 7f1a5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.977+0000 7f1a5d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:41.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.977+0000 7f1a5effd640 4 set_mon_vals no callback set 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.977+0000 7f1a5d7fa640 10 monclient: dump: 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:41.960 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:41.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.978+0000 7f1a65bad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:41.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.978+0000 7f1a65bad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:41.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.978+0000 7f1a65bad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:41.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.978+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a40005080 con 0x7f1a58160550 2024-03-29T10:59:41.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.979+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f1a60067bf0 con 0x7f1a58160550 2024-03-29T10:59:41.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.980+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f1a600cf2e0 con 0x7f1a58160550 2024-03-29T10:59:41.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.980+0000 7f1a5d7fa640 10 client.4490.objecter ms_dispatch 0x7f1a60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:41.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.980+0000 7f1a5d7fa640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:41.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.980+0000 7f1a5d7fa640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:41.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.980+0000 7f1a5d7fa640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T10:59:41.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.981+0000 7f1a64bab640 10 monclient: get_auth_request con 0x7f1a4406da70 auth_method 0 2024-03-29T10:59:41.962 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.981+0000 7f1a5d7fa640 10 client.4490.objecter ms_handle_connect 0x7f1a4406da70 2024-03-29T10:59:41.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.988+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1a600cae50 con 0x7f1a58160550 2024-03-29T10:59:41.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.988+0000 7f1a5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:41.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:41.988+0000 7f1a5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:42.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.346+0000 7f1a65bad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:42.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.346+0000 7f1a65bad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:42.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.346+0000 7f1a65bad640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:42.327 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.346+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a40005330 con 0x7f1a58160550 2024-03-29T10:59:42.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.350+0000 7f1a5d7fa640 1 -- 172.21.15.22:0/52425276 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f1a600ccd80 con 0x7f1a58160550 2024-03-29T10:59:42.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.350+0000 7f1a5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:42.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.350+0000 7f1a5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:42.331 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:42.331 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:42.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 10 librados: watch_flush enter 2024-03-29T10:59:42.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 10 librados: watch_flush exit 2024-03-29T10:59:42.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T10:59:42.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 10 client.4490.objecter successfully canceled tick 2024-03-29T10:59:42.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 >> v1:172.21.15.22:6816/380221644 conn(0x7f1a4406da70 legacy=0x7f1a4406ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 10 monclient: shutdown 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 >> v1:172.21.15.22:6790/0 conn(0x7f1a58160550 legacy=0x7f1a58167220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 shutdown_connections 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.354+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 >> 172.21.15.22:0/52425276 conn(0x7f1a58005c00 msgr2=0x7f1a580c4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.355+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 shutdown_connections 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.355+0000 7f1a65bad640 1 -- 172.21.15.22:0/52425276 wait complete. 2024-03-29T10:59:42.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:42.355+0000 7f1a65bad640 1 librados: shutdown 2024-03-29T10:59:42.350 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:49.157 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:49.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.335+0000 7f37e2786640 10 monclient: get_monmap_and_config 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.335+0000 7f37e2786640 10 monclient: build_initial_monmap 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.335+0000 7f37e2786640 10 monclient: monmap: 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:49.337382+0000 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:49.337382+0000 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:49.317 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:49.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.337+0000 7f37e2786640 1 Processor -- start 2024-03-29T10:59:49.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.338+0000 7f37e2786640 1 -- start start 2024-03-29T10:59:49.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.338+0000 7f37e2786640 10 monclient: init 2024-03-29T10:59:49.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.339+0000 7f37e2786640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:49.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.339+0000 7f37e2786640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:49.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.339+0000 7f37e2786640 10 monclient(hunting): picked mon.noname-c con 0x7f37dc0a9750 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.339+0000 7f37e2786640 10 monclient(hunting): picked mon.noname-a con 0x7f37dc0aad20 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.341+0000 7f37e2786640 10 monclient(hunting): picked mon.noname-b con 0x7f37dc0a9b30 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.341+0000 7f37e2786640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc0af4d0 con 0x7f37dc0aad20 2024-03-29T10:59:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.341+0000 7f37e2786640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc0af7b0 con 0x7f37dc0a9750 2024-03-29T10:59:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.341+0000 7f37e2786640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc0acab0 con 0x7f37dc0a9b30 2024-03-29T10:59:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.341+0000 7f37e2786640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:49.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.341+0000 7f37e2786640 10 monclient(hunting): authenticate will time out at 1090.646236s 2024-03-29T10:59:49.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.342+0000 7f37e1f85640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f37dc0a9b30 0x7f37dc149930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:38586/0 (socket says 172.21.15.22:38586) 2024-03-29T10:59:49.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.342+0000 7f37e1f85640 1 -- 172.21.15.22:0/2147146552 learned_addr learned my addr 172.21.15.22:0/2147146552 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391871485 0 0) 0x7f37dc0af7b0 con 0x7f37dc0a9750 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 10 monclient(hunting): my global_id is 4387 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c8003520 con 0x7f37dc0a9750 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1916272045 0 0) 0x7f37dc0acab0 con 0x7f37dc0a9b30 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.348+0000 7f37d3fff640 10 monclient(hunting): _negotiate 2024-03-29T10:59:49.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:49.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:49.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient(hunting): my global_id is 4493 2024-03-29T10:59:49.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37dc0af7b0 con 0x7f37dc0a9b30 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1916585122 0 0) 0x7f37dc0af7b0 con 0x7f37dc0a9b30 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 >> v1:172.21.15.26:6789/0 conn(0x7f37dc0a9750 legacy=0x7f37dc0ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 >> v1:172.21.15.22:6789/0 conn(0x7f37dc0aad20 legacy=0x7f37dc141a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient: _finish_hunting 0 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 1 monclient: found mon.noname-b 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc0aba30 con 0x7f37dc0a9b30 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient: _finish_auth 0 2024-03-29T10:59:49.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient: _check_auth_tickets 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f37d80015f0 con 0x7f37dc0a9b30 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.349+0000 7f37d3fff640 10 monclient: dump: 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:49.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37e2786640 5 monclient: authenticate success, global_id 4493 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37e2786640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37d8001750 con 0x7f37dc0a9b30 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37d3fff640 1 -- 172.21.15.22:0/2147146552 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f37d8001ae0 con 0x7f37dc0a9b30 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37e2786640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37e2786640 4 set_mon_vals no callback set 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37d3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.350+0000 7f37d3fff640 10 monclient: dump: 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.334 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.351+0000 7f37e2786640 10 monclient: shutdown 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.351+0000 7f37e2786640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.351+0000 7f37e2786640 1 -- 172.21.15.22:0/2147146552 >> v1:172.21.15.22:6790/0 conn(0x7f37dc0a9b30 legacy=0x7f37dc149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.351+0000 7f37e2786640 1 -- 172.21.15.22:0/2147146552 shutdown_connections 2024-03-29T10:59:49.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.351+0000 7f37e2786640 1 -- 172.21.15.22:0/2147146552 >> 172.21.15.22:0/2147146552 conn(0x7f37dc0673e0 msgr2=0x7f37dc0b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:49.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.352+0000 7f37e2786640 1 -- 172.21.15.22:0/2147146552 shutdown_connections 2024-03-29T10:59:49.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.353+0000 7f37e2786640 1 -- 172.21.15.22:0/2147146552 wait complete. 2024-03-29T10:59:49.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.354+0000 7f37e2786640 10 monclient: build_initial_monmap 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.354+0000 7f37e2786640 10 monclient: monmap: 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.354+0000 7f37e2786640 1 librados: starting msgr at 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.354+0000 7f37e2786640 1 librados: starting objecter 2024-03-29T10:59:49.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.354+0000 7f37e2786640 1 Processor -- start 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.354+0000 7f37e2786640 1 -- start start 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.354+0000 7f37e2786640 1 librados: setting wanted keys 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.355+0000 7f37e2786640 1 librados: calling monclient init 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.355+0000 7f37e2786640 10 monclient: init 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 10 monclient(hunting): picked mon.noname-b con 0x7f37dc0a9b30 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 10 monclient(hunting): picked mon.noname-a con 0x7f37dc0aad20 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 10 monclient(hunting): picked mon.noname-c con 0x7f37dc13b810 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e1f85640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f37dc13b810 0x7f37dc131f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39062/0 (socket says 172.21.15.22:39062) 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e1f85640 1 -- 172.21.15.22:0/2007204455 learned_addr learned my addr 172.21.15.22:0/2007204455 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc13a620 con 0x7f37dc0aad20 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc132610 con 0x7f37dc0a9b30 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37dc13a900 con 0x7f37dc13b810 2024-03-29T10:59:49.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.356+0000 7f37e2786640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.357+0000 7f37e2786640 10 monclient(hunting): authenticate will time out at 1090.662150s 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.357+0000 7f37d1ffb640 10 client.?.objecter ms_handle_connect 0x7f37dc13b810 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.357+0000 7f37d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.358+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171109258 0 0) 0x7f37dc13a900 con 0x7f37dc13b810 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.358+0000 7f37d1ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.358+0000 7f37d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.358+0000 7f37d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.358+0000 7f37d1ffb640 10 monclient(hunting): my global_id is 4496 2024-03-29T10:59:49.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.359+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c0004eb0 con 0x7f37dc13b810 2024-03-29T10:59:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1691644973 0 0) 0x7f37c0004eb0 con 0x7f37dc13b810 2024-03-29T10:59:49.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f37dc13a900 con 0x7f37dc13b810 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f37d80014b0 con 0x7f37dc13b810 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 10 monclient(hunting): dump: 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:49.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3805972432 0 0) 0x7f37dc13a900 con 0x7f37dc13b810 2024-03-29T10:59:49.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.360+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 >> v1:172.21.15.26:6789/0 conn(0x7f37dc0a9b30 legacy=0x7f37dc139670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:49.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 >> v1:172.21.15.22:6789/0 conn(0x7f37dc0aad20 legacy=0x7f37dc1317f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T10:59:49.345 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 1 monclient: found mon.c 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc13abe0 con 0x7f37dc13b810 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37e2786640 5 monclient: authenticate success, global_id 4496 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37e2786640 10 monclient: _renew_subs 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37e2786640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.363+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37dc13aec0 con 0x7f37dc13b810 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.364+0000 7f37e2786640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:49.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.364+0000 7f37e2786640 10 monclient: _renew_subs 2024-03-29T10:59:49.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.364+0000 7f37e2786640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:49.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.364+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f37dc139d80 con 0x7f37dc13b810 2024-03-29T10:59:49.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.364+0000 7f37e2786640 1 librados: init done 2024-03-29T10:59:49.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.366+0000 7f37e2786640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:49.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.366+0000 7f37e2786640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:49.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.366+0000 7f37e2786640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.366+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37dc138a40 con 0x7f37dc13b810 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.367+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37d80010f0 con 0x7f37dc13b810 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.367+0000 7f37d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.367+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f37d8004c30 con 0x7f37dc13b810 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.367+0000 7f37d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.367+0000 7f37d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.367+0000 7f37d1ffb640 10 monclient: dump: 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:49.349 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:49.350 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:49.350 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:49.350 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:49.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.367+0000 7f37d37fe640 4 set_mon_vals no callback set 2024-03-29T10:59:49.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.369+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f37d8004e50 con 0x7f37dc13b810 2024-03-29T10:59:49.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.370+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f37d8084ba0 con 0x7f37dc13b810 2024-03-29T10:59:49.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.370+0000 7f37d1ffb640 10 client.4496.objecter ms_dispatch 0x7f37dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:49.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.370+0000 7f37d1ffb640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:49.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.370+0000 7f37d1ffb640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:49.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.370+0000 7f37d1ffb640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-29T10:59:49.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.370+0000 7f37e1784640 10 monclient: get_auth_request con 0x7f37c006dbc0 auth_method 0 2024-03-29T10:59:49.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.371+0000 7f37d1ffb640 10 client.4496.objecter ms_handle_connect 0x7f37c006dbc0 2024-03-29T10:59:49.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.380+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f37d80504e0 con 0x7f37dc13b810 2024-03-29T10:59:49.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.381+0000 7f37d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:49.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.381+0000 7f37d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:49.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.752+0000 7f37e2786640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:49.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.752+0000 7f37e2786640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:49.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.752+0000 7f37e2786640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:49.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.753+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f37dc132870 con 0x7f37dc13b810 2024-03-29T10:59:49.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.754+0000 7f37d1ffb640 1 -- 172.21.15.22:0/2007204455 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f37dc132870 con 0x7f37dc13b810 2024-03-29T10:59:49.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.754+0000 7f37d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:49.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.754+0000 7f37d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:49.736 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:49.736 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:49.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.758+0000 7f37e2786640 10 librados: watch_flush enter 2024-03-29T10:59:49.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.758+0000 7f37e2786640 10 librados: watch_flush exit 2024-03-29T10:59:49.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.758+0000 7f37e2786640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-29T10:59:49.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.758+0000 7f37e2786640 10 client.4496.objecter successfully canceled tick 2024-03-29T10:59:49.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 >> v1:172.21.15.22:6816/380221644 conn(0x7f37c006dbc0 legacy=0x7f37c0070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:49.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 10 monclient: shutdown 2024-03-29T10:59:49.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:49.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 >> v1:172.21.15.22:6790/0 conn(0x7f37dc13b810 legacy=0x7f37dc131f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:49.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 shutdown_connections 2024-03-29T10:59:49.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 >> 172.21.15.22:0/2007204455 conn(0x7f37dc0673e0 msgr2=0x7f37dc0ac360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:49.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 shutdown_connections 2024-03-29T10:59:49.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 1 -- 172.21.15.22:0/2007204455 wait complete. 2024-03-29T10:59:49.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:49.759+0000 7f37e2786640 1 librados: shutdown 2024-03-29T10:59:49.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T10:59:56.566 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T10:59:56.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.744+0000 7f6551382640 10 monclient: get_monmap_and_config 2024-03-29T10:59:56.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.744+0000 7f6551382640 10 monclient: build_initial_monmap 2024-03-29T10:59:56.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.744+0000 7f6551382640 10 monclient: monmap: 2024-03-29T10:59:56.726 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:56.726 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:56.726 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:59:56.746721+0000 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:59:56.746721+0000 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.746+0000 7f6551382640 1 Processor -- start 2024-03-29T10:59:56.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.746+0000 7f6551382640 1 -- start start 2024-03-29T10:59:56.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.746+0000 7f6551382640 10 monclient: init 2024-03-29T10:59:56.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 10 monclient(hunting): picked mon.noname-a con 0x7f654c0ae250 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 10 monclient(hunting): picked mon.noname-c con 0x7f654c13f970 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 10 monclient(hunting): picked mon.noname-b con 0x7f654c142800 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c0afe00 con 0x7f654c0ae250 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c142c10 con 0x7f654c13f970 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c0afb20 con 0x7f654c142800 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.747+0000 7f6551382640 10 monclient(hunting): authenticate will time out at 1098.053045s 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.748+0000 7f654b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f654c142800 0x7f654c14da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53500/0 (socket says 172.21.15.22:53500) 2024-03-29T10:59:56.729 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.748+0000 7f654b7fe640 1 -- 172.21.15.22:0/2865991833 learned_addr learned my addr 172.21.15.22:0/2865991833 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:56.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493667156 0 0) 0x7f654c0afb20 con 0x7f654c142800 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): my global_id is 4499 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6534003520 con 0x7f654c142800 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030015924 0 0) 0x7f654c0afe00 con 0x7f654c0ae250 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): _negotiate 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient(hunting): my global_id is 4404 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c0afb20 con 0x7f654c0ae250 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2456990963 0 0) 0x7f6534003520 con 0x7f654c142800 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 >> v1:172.21.15.26:6789/0 conn(0x7f654c13f970 legacy=0x7f654c1420f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 >> v1:172.21.15.22:6789/0 conn(0x7f654c0ae250 legacy=0x7f654c13f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:56.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient: _finish_hunting 0 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 monclient: found mon.noname-b 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c0aab80 con 0x7f654c142800 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient: _finish_auth 0 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient: _check_auth_tickets 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6544001940 con 0x7f654c142800 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.749+0000 7f6549ffb640 10 monclient: dump: 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.733 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6551382640 5 monclient: authenticate success, global_id 4499 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6551382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6544001aa0 con 0x7f654c142800 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6549ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6549ffb640 1 -- 172.21.15.22:0/2865991833 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6544001eb0 con 0x7f654c142800 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6549ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6549ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6549ffb640 10 monclient: dump: 2024-03-29T10:59:56.734 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6551382640 10 monclient: get_monmap_and_config success 2024-03-29T10:59:56.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.750+0000 7f6551382640 4 set_mon_vals no callback set 2024-03-29T10:59:56.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.751+0000 7f6551382640 10 monclient: shutdown 2024-03-29T10:59:56.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.751+0000 7f6551382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:56.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.751+0000 7f6551382640 1 -- 172.21.15.22:0/2865991833 >> v1:172.21.15.22:6790/0 conn(0x7f654c142800 legacy=0x7f654c14da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:56.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.751+0000 7f6551382640 1 -- 172.21.15.22:0/2865991833 shutdown_connections 2024-03-29T10:59:56.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.751+0000 7f6551382640 1 -- 172.21.15.22:0/2865991833 >> 172.21.15.22:0/2865991833 conn(0x7f654c0b35e0 msgr2=0x7f654c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:56.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.752+0000 7f6551382640 1 -- 172.21.15.22:0/2865991833 shutdown_connections 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.752+0000 7f6551382640 1 -- 172.21.15.22:0/2865991833 wait complete. 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.753+0000 7f6551382640 10 monclient: build_initial_monmap 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.753+0000 7f6551382640 10 monclient: monmap: 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.753+0000 7f6551382640 1 librados: starting msgr at 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.753+0000 7f6551382640 1 librados: starting objecter 2024-03-29T10:59:56.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 1 Processor -- start 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 1 -- start start 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 1 librados: setting wanted keys 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 1 librados: calling monclient init 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient: init 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient: _reopen_session rank -1 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient(hunting): picked mon.noname-b con 0x7f654c0ae250 addr v1:172.21.15.26:6789/0 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient(hunting): picked mon.noname-c con 0x7f654c13f970 addr v1:172.21.15.22:6790/0 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient(hunting): picked mon.noname-a con 0x7f654c142800 addr v1:172.21.15.22:6789/0 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c1450d0 con 0x7f654c142800 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c1df560 con 0x7f654c0ae250 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f654c146090 con 0x7f654c13f970 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient(hunting): _renew_subs 2024-03-29T10:59:56.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.754+0000 7f6551382640 10 monclient(hunting): authenticate will time out at 1098.060028s 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.756+0000 7f654a7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f654c13f970 0x7f654c071c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53600/0 (socket says 172.21.15.22:53600) 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f654a7fc640 1 -- 172.21.15.22:0/772831218 learned_addr learned my addr 172.21.15.22:0/772831218 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 client.?.objecter ms_handle_connect 0x7f654c13f970 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 client.?.objecter ms_handle_connect 0x7f654c0ae250 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3947962022 0 0) 0x7f654c146090 con 0x7f654c13f970 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:56.743 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 monclient(hunting): my global_id is 4502 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6538004eb0 con 0x7f654c13f970 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 client.?.objecter ms_handle_connect 0x7f654c142800 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.757+0000 7f65337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710775328 0 0) 0x7f654c1df560 con 0x7f654c0ae250 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): my global_id is 4390 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c146090 con 0x7f654c0ae250 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 955801045 0 0) 0x7f6538004eb0 con 0x7f654c13f970 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f654c1df560 con 0x7f654c13f970 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f653c001870 con 0x7f654c13f970 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): dump: 2024-03-29T10:59:56.744 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.758+0000 7f65337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1374561589 0 0) 0x7f654c1450d0 con 0x7f654c142800 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient(hunting): _negotiate 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T10:59:56.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient(hunting): my global_id is 4407 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6538004eb0 con 0x7f654c142800 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 56468891 0 0) 0x7f654c146090 con 0x7f654c0ae250 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f654c1450d0 con 0x7f654c0ae250 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 820488632 0 0) 0x7f654c1df560 con 0x7f654c13f970 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 >> v1:172.21.15.26:6789/0 conn(0x7f654c0ae250 legacy=0x7f654c1dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 >> v1:172.21.15.22:6789/0 conn(0x7f654c142800 legacy=0x7f654c1e7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient: _finish_hunting 0 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 monclient: found mon.c 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f654c144c60 con 0x7f654c13f970 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient: _finish_auth 0 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 10 monclient: _check_auth_tickets 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f65337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f6551382640 5 monclient: authenticate success, global_id 4502 2024-03-29T10:59:56.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f6551382640 10 monclient: _renew_subs 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f6551382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.759+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f654c072340 con 0x7f654c13f970 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f6551382640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f6551382640 10 monclient: _renew_subs 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f6551382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f654c139210 con 0x7f654c13f970 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f6551382640 1 librados: init done 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f653c001d80 con 0x7f654c13f970 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f65337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f653c004aa0 con 0x7f654c13f970 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f65337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f65337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f65337fe640 10 monclient: dump: 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.747 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.760+0000 7f6548ff9640 4 set_mon_vals no callback set 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.762+0000 7f6551382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.762+0000 7f6551382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.762+0000 7f6551382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:56.748 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.762+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f652c005080 con 0x7f654c13f970 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.764+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f653c01aa70 con 0x7f654c13f970 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.765+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f653c084a70 con 0x7f654c13f970 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.765+0000 7f65337fe640 10 client.4502.objecter ms_dispatch 0x7f654c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.765+0000 7f65337fe640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.765+0000 7f65337fe640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.765+0000 7f654affd640 10 monclient: get_auth_request con 0x7f653806df70 auth_method 0 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.765+0000 7f65337fe640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-29T10:59:56.749 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.766+0000 7f65337fe640 10 client.4502.objecter ms_handle_connect 0x7f653806df70 2024-03-29T10:59:56.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.773+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f653c050180 con 0x7f654c13f970 2024-03-29T10:59:56.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.773+0000 7f65337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T10:59:56.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:56.773+0000 7f65337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T10:59:57.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.169+0000 7f6551382640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:57.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.169+0000 7f6551382640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:57.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.169+0000 7f6551382640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T10:59:57.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.169+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f652c005330 con 0x7f654c13f970 2024-03-29T10:59:57.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.171+0000 7f65337fe640 1 -- 172.21.15.22:0/772831218 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f653c052280 con 0x7f654c13f970 2024-03-29T10:59:57.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.171+0000 7f65337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T10:59:57.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.171+0000 7f65337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T10:59:57.153 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T10:59:57.153 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T10:59:57.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 10 librados: watch_flush enter 2024-03-29T10:59:57.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 10 librados: watch_flush exit 2024-03-29T10:59:57.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-29T10:59:57.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 10 client.4502.objecter successfully canceled tick 2024-03-29T10:59:57.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 >> v1:172.21.15.22:6816/380221644 conn(0x7f653806df70 legacy=0x7f6538070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:57.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 10 monclient: shutdown 2024-03-29T10:59:57.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T10:59:57.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.175+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 >> v1:172.21.15.22:6790/0 conn(0x7f654c13f970 legacy=0x7f654c071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T10:59:57.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.176+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 shutdown_connections 2024-03-29T10:59:57.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.176+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 >> 172.21.15.22:0/772831218 conn(0x7f654c0b35e0 msgr2=0x7f654c13cdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T10:59:57.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.176+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 shutdown_connections 2024-03-29T10:59:57.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.176+0000 7f6551382640 1 -- 172.21.15.22:0/772831218 wait complete. 2024-03-29T10:59:57.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T10:59:57.176+0000 7f6551382640 1 librados: shutdown 2024-03-29T10:59:57.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:03.981 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:04.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.138+0000 7f27d3a50640 10 monclient: get_monmap_and_config 2024-03-29T11:00:04.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.138+0000 7f27d3a50640 10 monclient: build_initial_monmap 2024-03-29T11:00:04.119 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.138+0000 7f27d3a50640 10 monclient: monmap: 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:04.140558+0000 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:04.140558+0000 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:04.120 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:04.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.139+0000 7f27d3a50640 1 Processor -- start 2024-03-29T11:00:04.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.139+0000 7f27d3a50640 1 -- start start 2024-03-29T11:00:04.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.139+0000 7f27d3a50640 10 monclient: init 2024-03-29T11:00:04.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.140+0000 7f27d3a50640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:04.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.140+0000 7f27d3a50640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.140+0000 7f27d3a50640 10 monclient(hunting): picked mon.noname-b con 0x7f27cc1422b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.140+0000 7f27d3a50640 10 monclient(hunting): picked mon.noname-a con 0x7f27cc142690 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d17ec640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f27cc1422b0 0x7f27cc13f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:51560/0 (socket says 172.21.15.22:51560) 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d17ec640 1 -- 172.21.15.22:0/1236670035 learned_addr learned my addr 172.21.15.22:0/1236670035 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d3a50640 10 monclient(hunting): picked mon.noname-c con 0x7f27cc0b17d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc142c80 con 0x7f27cc142690 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc0aae40 con 0x7f27cc0b17d0 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc13fbd0 con 0x7f27cc1422b0 2024-03-29T11:00:04.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d3a50640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:04.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27d3a50640 10 monclient(hunting): authenticate will time out at 1105.446720s 2024-03-29T11:00:04.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.141+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519174132 0 0) 0x7f27cc13fbd0 con 0x7f27cc1422b0 2024-03-29T11:00:04.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:04.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:04.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:04.123 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b8003520 con 0x7f27cc1422b0 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 346959225 0 0) 0x7f27cc0aae40 con 0x7f27cc0b17d0 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc13fbd0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2879105347 0 0) 0x7f27cc13fbd0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 >> v1:172.21.15.22:6790/0 conn(0x7f27cc1422b0 legacy=0x7f27cc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:04.124 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.142+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 >> v1:172.21.15.22:6789/0 conn(0x7f27cc142690 legacy=0x7f27cc0b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 1 monclient: found mon.noname-c 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc151940 con 0x7f27cc0b17d0 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c40019f0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: dump: 2024-03-29T11:00:04.127 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27c4001c10 con 0x7f27cc0b17d0 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.143+0000 7f27c3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27c3fff640 1 -- 172.21.15.22:0/1236670035 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c4001f20 con 0x7f27cc0b17d0 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27d3a50640 5 monclient: authenticate success, global_id 4393 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27d3a50640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27d3a50640 4 set_mon_vals no callback set 2024-03-29T11:00:04.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27c3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27c3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27c3fff640 10 monclient: dump: 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27d3a50640 10 monclient: shutdown 2024-03-29T11:00:04.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27d3a50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.144+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 >> v1:172.21.15.26:6789/0 conn(0x7f27cc0b17d0 legacy=0x7f27cc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.145+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 shutdown_connections 2024-03-29T11:00:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.145+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 >> 172.21.15.22:0/1236670035 conn(0x7f27cc066a00 msgr2=0x7f27cc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.145+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 shutdown_connections 2024-03-29T11:00:04.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.145+0000 7f27d3a50640 1 -- 172.21.15.22:0/1236670035 wait complete. 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.146+0000 7f27d3a50640 10 monclient: build_initial_monmap 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.146+0000 7f27d3a50640 10 monclient: monmap: 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:04.133 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.147+0000 7f27d3a50640 1 librados: starting msgr at 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.147+0000 7f27d3a50640 1 librados: starting objecter 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.147+0000 7f27d3a50640 1 Processor -- start 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.147+0000 7f27d3a50640 1 -- start start 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.147+0000 7f27d3a50640 1 librados: setting wanted keys 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.147+0000 7f27d3a50640 1 librados: calling monclient init 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.147+0000 7f27d3a50640 10 monclient: init 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.148+0000 7f27d3a50640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.148+0000 7f27d3a50640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:04.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.148+0000 7f27d3a50640 10 monclient(hunting): picked mon.noname-c con 0x7f27cc0b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.148+0000 7f27d3a50640 10 monclient(hunting): picked mon.noname-b con 0x7f27cc1422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.148+0000 7f27d17ec640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f27cc0b17d0 0x7f27cc1309b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:51678/0 (socket says 172.21.15.22:51678) 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d3a50640 10 monclient(hunting): picked mon.noname-a con 0x7f27cc142690 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d3a50640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc1324c0 con 0x7f27cc142690 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d3a50640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc14e1e0 con 0x7f27cc1422b0 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d3a50640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc0ac930 con 0x7f27cc0b17d0 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d3a50640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d3a50640 10 monclient(hunting): authenticate will time out at 1105.454590s 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d0feb640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f27cc1422b0 0x7f27cc14d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:42830/0 (socket says 172.21.15.22:42830) 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27d0feb640 1 -- 172.21.15.22:0/2004588961 learned_addr learned my addr 172.21.15.22:0/2004588961 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27c1ffb640 10 client.?.objecter ms_handle_connect 0x7f27cc1422b0 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.149+0000 7f27c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 client.?.objecter ms_handle_connect 0x7f27cc0b17d0 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:04.135 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383186554 0 0) 0x7f27cc0ac930 con 0x7f27cc0b17d0 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b0004eb0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271304318 0 0) 0x7f27cc14e1e0 con 0x7f27cc1422b0 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27cc0ac930 con 0x7f27cc1422b0 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4035602080 0 0) 0x7f27b0004eb0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27b0005490 con 0x7f27cc0b17d0 2024-03-29T11:00:04.136 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c80012c0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:04.137 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2166647119 0 0) 0x7f27cc0ac930 con 0x7f27cc1422b0 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27b0004eb0 con 0x7f27cc1422b0 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27bc001750 con 0x7f27cc1422b0 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:04.138 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1942294471 0 0) 0x7f27b0005490 con 0x7f27cc0b17d0 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 >> v1:172.21.15.26:6789/0 conn(0x7f27cc1422b0 legacy=0x7f27cc14d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.150+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 >> v1:172.21.15.22:6789/0 conn(0x7f27cc142690 legacy=0x7f27cc14dad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 1 monclient: found mon.c 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0acc10 con 0x7f27cc0b17d0 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:04.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27d3a50640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27c80047e0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f27c80016d0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c1ffb640 10 monclient: dump: 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:04.140 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.151+0000 7f27c37fe640 4 set_mon_vals no callback set 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 10 monclient: _renew_subs 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc132950 con 0x7f27cc0b17d0 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 10 monclient: _renew_subs 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc1312a0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.152+0000 7f27d3a50640 1 librados: init done 2024-03-29T11:00:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.154+0000 7f27d3a50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.154+0000 7f27d3a50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.154+0000 7f27d3a50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.154+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc075f10 con 0x7f27cc0b17d0 2024-03-29T11:00:04.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.155+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f27c801a960 con 0x7f27cc0b17d0 2024-03-29T11:00:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.156+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f27c80849a0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.156+0000 7f27c1ffb640 10 client.4508.objecter ms_dispatch 0x7f27cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.156+0000 7f27c1ffb640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.156+0000 7f27c1ffb640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.156+0000 7f27c1ffb640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-29T11:00:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.156+0000 7f27d0feb640 10 monclient: get_auth_request con 0x7f27b006e1c0 auth_method 0 2024-03-29T11:00:04.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.157+0000 7f27c1ffb640 10 client.4508.objecter ms_handle_connect 0x7f27b006e1c0 2024-03-29T11:00:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.163+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27c8050280 con 0x7f27cc0b17d0 2024-03-29T11:00:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.163+0000 7f27c1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:04.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.163+0000 7f27c1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:04.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.475+0000 7f27d3a50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:04.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.475+0000 7f27d3a50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:04.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.475+0000 7f27d3a50640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:04.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.475+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27cc0b2270 con 0x7f27cc0b17d0 2024-03-29T11:00:04.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.476+0000 7f27c1ffb640 1 -- 172.21.15.22:0/2004588961 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f27c80521b0 con 0x7f27cc0b17d0 2024-03-29T11:00:04.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.476+0000 7f27c1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:04.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.476+0000 7f27c1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:04.457 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:04.457 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:04.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 10 librados: watch_flush enter 2024-03-29T11:00:04.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 10 librados: watch_flush exit 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 10 client.4508.objecter successfully canceled tick 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 >> v1:172.21.15.22:6816/380221644 conn(0x7f27b006e1c0 legacy=0x7f27b0070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 10 monclient: shutdown 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.479+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 >> v1:172.21.15.22:6790/0 conn(0x7f27cc0b17d0 legacy=0x7f27cc1309b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.480+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 shutdown_connections 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.480+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 >> 172.21.15.22:0/2004588961 conn(0x7f27cc066a00 msgr2=0x7f27cc0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:04.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.480+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 shutdown_connections 2024-03-29T11:00:04.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.480+0000 7f27d3a50640 1 -- 172.21.15.22:0/2004588961 wait complete. 2024-03-29T11:00:04.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:04.480+0000 7f27d3a50640 1 librados: shutdown 2024-03-29T11:00:04.475 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:11.282 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.438+0000 7fb1291ca640 10 monclient: get_monmap_and_config 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.438+0000 7fb1291ca640 10 monclient: build_initial_monmap 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.438+0000 7fb1291ca640 10 monclient: monmap: 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:11.438940+0000 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:11.438940+0000 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:11.418 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:11.419 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.438+0000 7fb1291ca640 1 Processor -- start 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.438+0000 7fb1291ca640 1 -- start start 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.438+0000 7fb1291ca640 10 monclient: init 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 10 monclient(hunting): picked mon.noname-b con 0x7fb1241457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 10 monclient(hunting): picked mon.noname-a con 0x7fb12413f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 10 monclient(hunting): picked mon.noname-c con 0x7fb1240a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1240aaef0 con 0x7fb12413f650 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124145bd0 con 0x7fb1240a9d70 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124145eb0 con 0x7fb1241457f0 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:11.422 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.439+0000 7fb1291ca640 10 monclient(hunting): authenticate will time out at 1112.743903s 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb122d9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb1241457f0 0x7fb12413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40206/0 (socket says 172.21.15.22:40206) 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb122d9d640 1 -- 172.21.15.22:0/1156549731 learned_addr learned my addr 172.21.15.22:0/1156549731 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3838875882 0 0) 0x7fb124145eb0 con 0x7fb1241457f0 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb121d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb121d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb121d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb121d9b640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.440+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb10c003520 con 0x7fb1241457f0 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3432350268 0 0) 0x7fb124145bd0 con 0x7fb1240a9d70 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb124145eb0 con 0x7fb1240a9d70 2024-03-29T11:00:11.423 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2857665081 0 0) 0x7fb10c003520 con 0x7fb1241457f0 2024-03-29T11:00:11.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 >> v1:172.21.15.26:6789/0 conn(0x7fb1240a9d70 legacy=0x7fb12414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:11.424 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.441+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 >> v1:172.21.15.22:6789/0 conn(0x7fb12413f650 legacy=0x7fb1240b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:11.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:00:11.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 1 monclient: found mon.noname-b 2024-03-29T11:00:11.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:11.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:00:11.426 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124146340 con 0x7fb1241457f0 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 10 monclient: _finish_auth 0 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.443+0000 7fb121d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb114001690 con 0x7fb1241457f0 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 10 monclient: dump: 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:11.427 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb1140018b0 con 0x7fb1241457f0 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 1 -- 172.21.15.22:0/1156549731 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb114001c40 con 0x7fb1241457f0 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb121d9b640 10 monclient: dump: 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:11.428 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb1291ca640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb1291ca640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.444+0000 7fb1291ca640 4 set_mon_vals no callback set 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.445+0000 7fb1291ca640 10 monclient: shutdown 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.445+0000 7fb1291ca640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.445+0000 7fb1291ca640 1 -- 172.21.15.22:0/1156549731 >> v1:172.21.15.22:6790/0 conn(0x7fb1241457f0 legacy=0x7fb12413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.446+0000 7fb1291ca640 1 -- 172.21.15.22:0/1156549731 shutdown_connections 2024-03-29T11:00:11.429 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.446+0000 7fb1291ca640 1 -- 172.21.15.22:0/1156549731 >> 172.21.15.22:0/1156549731 conn(0x7fb124066a00 msgr2=0x7fb124066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:11.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.446+0000 7fb1291ca640 1 -- 172.21.15.22:0/1156549731 shutdown_connections 2024-03-29T11:00:11.430 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.446+0000 7fb1291ca640 1 -- 172.21.15.22:0/1156549731 wait complete. 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.447+0000 7fb1291ca640 10 monclient: build_initial_monmap 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.447+0000 7fb1291ca640 10 monclient: monmap: 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:11.432 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.447+0000 7fb1291ca640 1 librados: starting msgr at 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.447+0000 7fb1291ca640 1 librados: starting objecter 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.447+0000 7fb1291ca640 1 Processor -- start 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.448+0000 7fb1291ca640 1 -- start start 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.448+0000 7fb1291ca640 1 librados: setting wanted keys 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.448+0000 7fb1291ca640 1 librados: calling monclient init 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.448+0000 7fb1291ca640 10 monclient: init 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.448+0000 7fb1291ca640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.449+0000 7fb1291ca640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:11.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.449+0000 7fb1291ca640 10 monclient(hunting): picked mon.noname-b con 0x7fb1240a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.449+0000 7fb1291ca640 10 monclient(hunting): picked mon.noname-a con 0x7fb12413f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb1291ca640 10 monclient(hunting): picked mon.noname-c con 0x7fb1241457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb1291ca640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1240709d0 con 0x7fb12413f650 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb1291ca640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124072000 con 0x7fb1240a9d70 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb1291ca640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb124135f00 con 0x7fb1241457f0 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb1291ca640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb1291ca640 10 monclient(hunting): authenticate will time out at 1112.754160s 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb12359e640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb1241457f0 0x7fb1241357f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40472/0 (socket says 172.21.15.22:40472) 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb12359e640 1 -- 172.21.15.22:0/3837521239 learned_addr learned my addr 172.21.15.22:0/3837521239 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 client.?.objecter ms_handle_connect 0x7fb1241457f0 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156960454 0 0) 0x7fb124135f00 con 0x7fb1241457f0 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:11.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): my global_id is 4514 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb110004eb0 con 0x7fb1241457f0 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4121665559 0 0) 0x7fb110004eb0 con 0x7fb1241457f0 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb124135f00 con 0x7fb1241457f0 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb11c001dd0 con 0x7fb1241457f0 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): dump: 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:11.435 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.450+0000 7fb10b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 10 client.?.objecter ms_handle_connect 0x7fb1240a9d70 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3510905313 0 0) 0x7fb124135f00 con 0x7fb1241457f0 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 >> v1:172.21.15.26:6789/0 conn(0x7fb1240a9d70 legacy=0x7fb1241db670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 >> v1:172.21.15.22:6789/0 conn(0x7fb12413f650 legacy=0x7fb1241350e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 1 monclient: found mon.c 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:11.436 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124070e60 con 0x7fb1241457f0 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb10b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 5 monclient: authenticate success, global_id 4514 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 10 monclient: _renew_subs 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 1 -- 172.21.15.22:0/3837521239 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1240703f0 con 0x7fb1241457f0 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 10 client.4514.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 10 monclient: _renew_subs 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 1 -- 172.21.15.22:0/3837521239 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1241424c0 con 0x7fb1241457f0 2024-03-29T11:00:11.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.451+0000 7fb1291ca640 1 librados: init done 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.452+0000 7fb1097fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.452+0000 7fb1097fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.452+0000 7fb1097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.452+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb104005080 con 0x7fb1241457f0 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb11c002030 con 0x7fb1241457f0 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb10b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb11c003dc0 con 0x7fb1241457f0 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb10b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb10b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb10b7fe640 10 monclient: dump: 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:11.439 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fb11c003fe0 con 0x7fb1241457f0 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.453+0000 7fb120d99640 4 set_mon_vals no callback set 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.455+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fb11c084ef0 con 0x7fb1241457f0 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.455+0000 7fb10b7fe640 10 client.4514.objecter ms_dispatch 0x7fb124000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.455+0000 7fb10b7fe640 3 client.4514.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.455+0000 7fb10b7fe640 3 client.4514.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:11.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.455+0000 7fb10b7fe640 20 client.4514.objecter dump_active .. 0 homeless 2024-03-29T11:00:11.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.455+0000 7fb122d9d640 10 monclient: get_auth_request con 0x7fb11006dc60 auth_method 0 2024-03-29T11:00:11.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.455+0000 7fb10b7fe640 10 client.4514.objecter ms_handle_connect 0x7fb11006dc60 2024-03-29T11:00:11.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.460+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb11c0507d0 con 0x7fb1241457f0 2024-03-29T11:00:11.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.461+0000 7fb10b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:11.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.461+0000 7fb10b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:11.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.770+0000 7fb1097fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:11.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.770+0000 7fb1097fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:11.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.770+0000 7fb1097fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:11.750 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.770+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb104005330 con 0x7fb1241457f0 2024-03-29T11:00:11.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.771+0000 7fb10b7fe640 1 -- 172.21.15.22:0/3837521239 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fb11c052700 con 0x7fb1241457f0 2024-03-29T11:00:11.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.771+0000 7fb10b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:11.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.771+0000 7fb10b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:11.751 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:11.751 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:11.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 10 librados: watch_flush enter 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 10 librados: watch_flush exit 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 20 client.4514.objecter shutdown clearing up homeless session... 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 10 client.4514.objecter successfully canceled tick 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 >> v1:172.21.15.22:6816/380221644 conn(0x7fb11006dc60 legacy=0x7fb110070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 10 monclient: shutdown 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 >> v1:172.21.15.22:6790/0 conn(0x7fb1241457f0 legacy=0x7fb1241357f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 shutdown_connections 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 >> 172.21.15.22:0/3837521239 conn(0x7fb124066a00 msgr2=0x7fb1240b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.773+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 shutdown_connections 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.774+0000 7fb1097fa640 1 -- 172.21.15.22:0/3837521239 wait complete. 2024-03-29T11:00:11.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:11.774+0000 7fb1097fa640 1 librados: shutdown 2024-03-29T11:00:11.768 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:18.575 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.745+0000 7f7ae74a2640 10 monclient: get_monmap_and_config 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.745+0000 7f7ae74a2640 10 monclient: build_initial_monmap 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.745+0000 7f7ae74a2640 10 monclient: monmap: 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:18.745987+0000 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:18.745987+0000 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:18.725 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:18.726 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:18.726 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:18.726 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:18.726 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.746+0000 7f7ae74a2640 1 Processor -- start 2024-03-29T11:00:18.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.746+0000 7f7ae74a2640 1 -- start start 2024-03-29T11:00:18.727 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.746+0000 7f7ae74a2640 10 monclient: init 2024-03-29T11:00:18.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.747+0000 7f7ae74a2640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:18.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.747+0000 7f7ae74a2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:18.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.747+0000 7f7ae74a2640 10 monclient(hunting): picked mon.noname-b con 0x7f7ad80345e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:18.728 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.748+0000 7f7ae74a2640 10 monclient(hunting): picked mon.noname-c con 0x7f7ad80349c0 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.749+0000 7f7ae74a2640 10 monclient(hunting): picked mon.noname-a con 0x7f7ad802b570 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.749+0000 7f7ae74a2640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad8032ef0 con 0x7f7ad802b570 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.749+0000 7f7ae74a2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad80be170 con 0x7f7ad80349c0 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.749+0000 7f7ae74a2640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad802c720 con 0x7f7ad80345e0 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.749+0000 7f7ae74a2640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.749+0000 7f7ae74a2640 10 monclient(hunting): authenticate will time out at 1120.054019s 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.750+0000 7f7ae64a0640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7ad80345e0 0x7f7ad80bda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57550/0 (socket says 172.21.15.22:57550) 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.750+0000 7f7ae64a0640 1 -- 172.21.15.22:0/2045894999 learned_addr learned my addr 172.21.15.22:0/2045894999 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:18.731 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.750+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2776157271 0 0) 0x7f7ad802c720 con 0x7f7ad80345e0 2024-03-29T11:00:18.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.750+0000 7f7ae549e640 10 monclient(hunting): _negotiate 2024-03-29T11:00:18.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.751+0000 7f7ae549e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:18.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.751+0000 7f7ae549e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:18.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.751+0000 7f7ae549e640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:00:18.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.751+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7acc003520 con 0x7f7ad80345e0 2024-03-29T11:00:18.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.751+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4001051014 0 0) 0x7f7acc003520 con 0x7f7ad80345e0 2024-03-29T11:00:18.732 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.751+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 >> v1:172.21.15.26:6789/0 conn(0x7f7ad80349c0 legacy=0x7f7ad80c7260 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:18.733 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.752+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 >> v1:172.21.15.22:6789/0 conn(0x7f7ad802b570 legacy=0x7f7ad80c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.752+0000 7f7ae549e640 10 monclient: _finish_hunting 0 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 1 monclient: found mon.noname-b 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad802c990 con 0x7f7ad80345e0 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 10 monclient: _finish_auth 0 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 10 monclient: _check_auth_tickets 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7ad0001690 con 0x7f7ad80345e0 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 10 monclient: dump: 2024-03-29T11:00:18.734 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae74a2640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae74a2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.753+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ad0001860 con 0x7f7ad80345e0 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.754+0000 7f7ae549e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.754+0000 7f7ae549e640 1 -- 172.21.15.22:0/2045894999 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7ad0001c70 con 0x7f7ad80345e0 2024-03-29T11:00:18.735 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.754+0000 7f7ae549e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.754+0000 7f7ae549e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.754+0000 7f7ae549e640 10 monclient: dump: 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:18.736 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:18.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.755+0000 7f7ae74a2640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:18.737 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.755+0000 7f7ae74a2640 4 set_mon_vals no callback set 2024-03-29T11:00:18.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.756+0000 7f7ae74a2640 10 monclient: shutdown 2024-03-29T11:00:18.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.756+0000 7f7ae74a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:18.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.756+0000 7f7ae74a2640 1 -- 172.21.15.22:0/2045894999 >> v1:172.21.15.22:6790/0 conn(0x7f7ad80345e0 legacy=0x7f7ad80bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:18.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.756+0000 7f7ae74a2640 1 -- 172.21.15.22:0/2045894999 shutdown_connections 2024-03-29T11:00:18.738 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.756+0000 7f7ae74a2640 1 -- 172.21.15.22:0/2045894999 >> 172.21.15.22:0/2045894999 conn(0x7f7ad80058f0 msgr2=0x7f7ad8005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:18.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.758+0000 7f7ae74a2640 1 -- 172.21.15.22:0/2045894999 shutdown_connections 2024-03-29T11:00:18.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.759+0000 7f7ae74a2640 1 -- 172.21.15.22:0/2045894999 wait complete. 2024-03-29T11:00:18.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.759+0000 7f7ae74a2640 10 monclient: build_initial_monmap 2024-03-29T11:00:18.739 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.759+0000 7f7ae74a2640 10 monclient: monmap: 2024-03-29T11:00:18.739 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:18.739 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:18.739 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.760+0000 7f7ae74a2640 1 librados: starting msgr at 2024-03-29T11:00:18.740 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.760+0000 7f7ae74a2640 1 librados: starting objecter 2024-03-29T11:00:18.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.760+0000 7f7ae74a2640 1 Processor -- start 2024-03-29T11:00:18.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.760+0000 7f7ae74a2640 1 -- start start 2024-03-29T11:00:18.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.760+0000 7f7ae74a2640 1 librados: setting wanted keys 2024-03-29T11:00:18.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.760+0000 7f7ae74a2640 1 librados: calling monclient init 2024-03-29T11:00:18.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.760+0000 7f7ae74a2640 10 monclient: init 2024-03-29T11:00:18.741 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.761+0000 7f7ae74a2640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:18.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.761+0000 7f7ae74a2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:18.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.761+0000 7f7ae74a2640 10 monclient(hunting): picked mon.noname-b con 0x7f7ad802b570 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:18.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.762+0000 7f7ae74a2640 10 monclient(hunting): picked mon.noname-c con 0x7f7ad80c0a20 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:18.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.762+0000 7f7ae5c9f640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7ad80c0a20 0x7f7ad815bce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57784/0 (socket says 172.21.15.22:57784) 2024-03-29T11:00:18.742 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.763+0000 7f7ae5c9f640 1 -- 172.21.15.22:0/3403640550 learned_addr learned my addr 172.21.15.22:0/3403640550 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:18.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.763+0000 7f7ae74a2640 10 monclient(hunting): picked mon.noname-a con 0x7f7ad80c5060 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:18.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.763+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad80c1320 con 0x7f7ad80c5060 2024-03-29T11:00:18.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.763+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad80c2460 con 0x7f7ad802b570 2024-03-29T11:00:18.745 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.763+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ad80c0f20 con 0x7f7ad80c0a20 2024-03-29T11:00:18.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.763+0000 7f7ae74a2640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:18.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.763+0000 7f7ae74a2640 10 monclient(hunting): authenticate will time out at 1120.068020s 2024-03-29T11:00:18.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.765+0000 7f7ad6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ad80c0a20 2024-03-29T11:00:18.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.765+0000 7f7ad6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:18.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.765+0000 7f7ad6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ad802b570 2024-03-29T11:00:18.746 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.765+0000 7f7ad6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995676451 0 0) 0x7f7ad80c0f20 con 0x7f7ad80c0a20 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ac4004eb0 con 0x7f7ad80c0a20 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983086369 0 0) 0x7f7ad80c2460 con 0x7f7ad802b570 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.766+0000 7f7ad6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ad80c0f20 con 0x7f7ad802b570 2024-03-29T11:00:18.751 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1593710788 0 0) 0x7f7ac4004eb0 con 0x7f7ad80c0a20 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ad80c2460 con 0x7f7ad80c0a20 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7adc0017d0 con 0x7f7ad80c0a20 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient(hunting): dump: 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:18.752 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1685916051 0 0) 0x7f7ad80c2460 con 0x7f7ad80c0a20 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 >> v1:172.21.15.26:6789/0 conn(0x7f7ad802b570 legacy=0x7f7ad80c1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 >> v1:172.21.15.22:6789/0 conn(0x7f7ad80c5060 legacy=0x7f7ad8163220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 1 monclient: found mon.c 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.767+0000 7f7ad6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad815c3f0 con 0x7f7ad80c0a20 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ae74a2640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:00:18.753 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7adc001ce0 con 0x7f7ad80c0a20 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ae74a2640 10 monclient: _renew_subs 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ae74a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ad8164940 con 0x7f7ad80c0a20 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ae74a2640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7adc004a00 con 0x7f7ad80c0a20 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.768+0000 7f7ae4c9d640 4 set_mon_vals no callback set 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.769+0000 7f7ad6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.769+0000 7f7ad6ffd640 10 monclient: dump: 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:18.754 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.769+0000 7f7ae74a2640 10 monclient: _renew_subs 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.769+0000 7f7ae74a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.769+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ad8164cb0 con 0x7f7ad80c0a20 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.769+0000 7f7ae74a2640 1 librados: init done 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.770+0000 7f7ae74a2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.770+0000 7f7ae74a2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.770+0000 7f7ae74a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:18.755 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.770+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ad80c3390 con 0x7f7ad80c0a20 2024-03-29T11:00:18.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.774+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f7adc01a9d0 con 0x7f7ad80c0a20 2024-03-29T11:00:18.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.775+0000 7f7ae64a0640 10 monclient: get_auth_request con 0x7f7ac406dd90 auth_method 0 2024-03-29T11:00:18.756 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.775+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f7adc084950 con 0x7f7ad80c0a20 2024-03-29T11:00:18.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.775+0000 7f7ad6ffd640 10 client.4520.objecter ms_dispatch 0x7f7ae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:18.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.775+0000 7f7ad6ffd640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:18.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.775+0000 7f7ad6ffd640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:18.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.775+0000 7f7ad6ffd640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-29T11:00:18.757 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.775+0000 7f7ad6ffd640 10 client.4520.objecter ms_handle_connect 0x7f7ac406dd90 2024-03-29T11:00:18.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.778+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7adc050060 con 0x7f7ad80c0a20 2024-03-29T11:00:18.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.778+0000 7f7ad6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:18.758 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:18.778+0000 7f7ad6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:19.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.119+0000 7f7ae74a2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:19.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.119+0000 7f7ae74a2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:19.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.119+0000 7f7ae74a2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:00:19.099 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.119+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ad8004cd0 con 0x7f7ad80c0a20 2024-03-29T11:00:19.101 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.122+0000 7f7ad6ffd640 1 -- 172.21.15.22:0/3403640550 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f7adc052160 con 0x7f7ad80c0a20 2024-03-29T11:00:19.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.122+0000 7f7ad6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:19.102 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.122+0000 7f7ad6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:19.102 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:19.103 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:19.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 10 librados: watch_flush enter 2024-03-29T11:00:19.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 10 librados: watch_flush exit 2024-03-29T11:00:19.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-29T11:00:19.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 10 client.4520.objecter successfully canceled tick 2024-03-29T11:00:19.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 >> v1:172.21.15.22:6816/380221644 conn(0x7f7ac406dd90 legacy=0x7f7ac4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:19.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 10 monclient: shutdown 2024-03-29T11:00:19.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:19.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 >> v1:172.21.15.22:6790/0 conn(0x7f7ad80c0a20 legacy=0x7f7ad815bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:19.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 shutdown_connections 2024-03-29T11:00:19.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.127+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 >> 172.21.15.22:0/3403640550 conn(0x7f7ad80058f0 msgr2=0x7f7ad8033d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:19.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.128+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 shutdown_connections 2024-03-29T11:00:19.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.128+0000 7f7ae74a2640 1 -- 172.21.15.22:0/3403640550 wait complete. 2024-03-29T11:00:19.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:19.129+0000 7f7ae74a2640 1 librados: shutdown 2024-03-29T11:00:19.126 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:25.933 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:26.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.101+0000 7fa5739e2640 10 monclient: get_monmap_and_config 2024-03-29T11:00:26.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.101+0000 7fa5739e2640 10 monclient: build_initial_monmap 2024-03-29T11:00:26.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.101+0000 7fa5739e2640 10 monclient: monmap: 2024-03-29T11:00:26.081 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:26.102425+0000 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:26.102425+0000 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:26.082 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:26.083 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.102+0000 7fa5739e2640 1 Processor -- start 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 1 -- start start 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 10 monclient: init 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa56c1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c13f650 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa56c0a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.103+0000 7fa5739e2640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c0aaef0 con 0x7fa56c1457f0 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.104+0000 7fa5739e2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c145bd0 con 0x7fa56c13f650 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.104+0000 7fa5739e2640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c145eb0 con 0x7fa56c0a9d70 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.104+0000 7fa5739e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.104+0000 7fa5739e2640 10 monclient(hunting): authenticate will time out at 1127.408102s 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.104+0000 7fa571f7f640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa56c0a9d70 0x7fa56c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34332/0 (socket says 172.21.15.22:34332) 2024-03-29T11:00:26.084 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.104+0000 7fa571f7f640 1 -- 172.21.15.22:0/3525192073 learned_addr learned my addr 172.21.15.22:0/3525192073 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:26.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.106+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048283479 0 0) 0x7fa56c145eb0 con 0x7fa56c0a9d70 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.106+0000 7fa563fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.106+0000 7fa563fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.106+0000 7fa563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.106+0000 7fa563fff640 10 monclient(hunting): my global_id is 4523 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.106+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa558003520 con 0x7fa56c0a9d70 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138982373 0 0) 0x7fa56c0aaef0 con 0x7fa56c1457f0 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient(hunting): _negotiate 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:00:26.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56c145eb0 con 0x7fa56c1457f0 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3852703377 0 0) 0x7fa56c145eb0 con 0x7fa56c1457f0 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 >> v1:172.21.15.22:6790/0 conn(0x7fa56c0a9d70 legacy=0x7fa56c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 >> v1:172.21.15.26:6789/0 conn(0x7fa56c13f650 legacy=0x7fa56c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient: _finish_hunting 0 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 monclient: found mon.noname-a 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c146340 con 0x7fa56c1457f0 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient: _finish_auth 0 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient: _check_auth_tickets 2024-03-29T11:00:26.090 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa5739e2640 5 monclient: authenticate success, global_id 4413 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa5739e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa5640019b0 con 0x7fa56c1457f0 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.107+0000 7fa563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 10 monclient: dump: 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:26.091 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa5739e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa564001c00 con 0x7fa56c1457f0 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa5739e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa5739e2640 4 set_mon_vals no callback set 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 1 -- 172.21.15.22:0/3525192073 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa564001f90 con 0x7fa56c1457f0 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:26.092 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.108+0000 7fa563fff640 10 monclient: dump: 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.109+0000 7fa5739e2640 10 monclient: shutdown 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.109+0000 7fa5739e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.109+0000 7fa5739e2640 1 -- 172.21.15.22:0/3525192073 >> v1:172.21.15.22:6789/0 conn(0x7fa56c1457f0 legacy=0x7fa56c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:26.093 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.109+0000 7fa5739e2640 1 -- 172.21.15.22:0/3525192073 shutdown_connections 2024-03-29T11:00:26.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.110+0000 7fa5739e2640 1 -- 172.21.15.22:0/3525192073 >> 172.21.15.22:0/3525192073 conn(0x7fa56c066a00 msgr2=0x7fa56c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:26.094 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.110+0000 7fa5739e2640 1 -- 172.21.15.22:0/3525192073 shutdown_connections 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.110+0000 7fa5739e2640 1 -- 172.21.15.22:0/3525192073 wait complete. 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 10 monclient: build_initial_monmap 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 10 monclient: monmap: 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 1 librados: starting msgr at 2024-03-29T11:00:26.095 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 1 librados: starting objecter 2024-03-29T11:00:26.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 1 Processor -- start 2024-03-29T11:00:26.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 1 -- start start 2024-03-29T11:00:26.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 1 librados: setting wanted keys 2024-03-29T11:00:26.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 1 librados: calling monclient init 2024-03-29T11:00:26.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.111+0000 7fa5739e2640 10 monclient: init 2024-03-29T11:00:26.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:26.096 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:26.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 10 monclient(hunting): picked mon.noname-b con 0x7fa56c0a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:26.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 10 monclient(hunting): picked mon.noname-c con 0x7fa56c13f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:26.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 10 monclient(hunting): picked mon.noname-a con 0x7fa56c1457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:26.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c138a90 con 0x7fa56c1457f0 2024-03-29T11:00:26.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c13a940 con 0x7fa56c0a9d70 2024-03-29T11:00:26.104 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c135800 con 0x7fa56c13f650 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.112+0000 7fa5739e2640 10 monclient(hunting): authenticate will time out at 1127.416953s 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.113+0000 7fa570f7d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa56c13f650 0x7fa56c1349e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34432/0 (socket says 172.21.15.22:34432) 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.113+0000 7fa570f7d640 1 -- 172.21.15.22:0/3832103533 learned_addr learned my addr 172.21.15.22:0/3832103533 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.113+0000 7fa561ffb640 10 client.?.objecter ms_handle_connect 0x7fa56c13f650 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.113+0000 7fa561ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 client.?.objecter ms_handle_connect 0x7fa56c1457f0 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117447121 0 0) 0x7fa56c138a90 con 0x7fa56c1457f0 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:00:26.105 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa550004eb0 con 0x7fa56c1457f0 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299231631 0 0) 0x7fa56c135800 con 0x7fa56c13f650 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa56c138a90 con 0x7fa56c13f650 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.116+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686107888 0 0) 0x7fa550004eb0 con 0x7fa56c1457f0 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa56c135800 con 0x7fa56c1457f0 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 client.?.objecter ms_handle_connect 0x7fa56c0a9d70 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa55c001950 con 0x7fa56c1457f0 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:26.106 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251742310 0 0) 0x7fa56c13a940 con 0x7fa56c0a9d70 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:26.107 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5500068a0 con 0x7fa56c0a9d70 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1511698163 0 0) 0x7fa56c138a90 con 0x7fa56c13f650 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.117+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa56c13a940 con 0x7fa56c13f650 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa568003fc0 con 0x7fa56c13f650 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 10 monclient(hunting): dump: 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:26.108 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 702420151 0 0) 0x7fa56c135800 con 0x7fa56c1457f0 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 >> v1:172.21.15.22:6790/0 conn(0x7fa56c13f650 legacy=0x7fa56c1349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 >> v1:172.21.15.26:6789/0 conn(0x7fa56c0a9d70 legacy=0x7fa56c1d79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 1 monclient: found mon.a 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c13abb0 con 0x7fa56c1457f0 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 10 monclient: _finish_auth 0 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa5739e2640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa5739e2640 10 monclient: _renew_subs 2024-03-29T11:00:26.109 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa5739e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.118+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c13ae90 con 0x7fa56c1457f0 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.119+0000 7fa5739e2640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.119+0000 7fa5739e2640 10 monclient: _renew_subs 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.119+0000 7fa5739e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.119+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c0b0d60 con 0x7fa56c1457f0 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.119+0000 7fa5739e2640 1 librados: init done 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.120+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa55c0047e0 con 0x7fa56c1457f0 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.120+0000 7fa561ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.120+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa55c001950 con 0x7fa56c1457f0 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.120+0000 7fa561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.120+0000 7fa561ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.120+0000 7fa561ffb640 10 monclient: dump: 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:26.110 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.121+0000 7fa5739e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.121+0000 7fa5739e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:26.111 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.121+0000 7fa5739e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.121+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fa55c01a7c0 con 0x7fa56c1457f0 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.121+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c075f10 con 0x7fa56c1457f0 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.121+0000 7fa5637fe640 4 set_mon_vals no callback set 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.122+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fa55c084650 con 0x7fa56c1457f0 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.122+0000 7fa561ffb640 10 client.4416.objecter ms_dispatch 0x7fa56c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.122+0000 7fa561ffb640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.122+0000 7fa561ffb640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.122+0000 7fa561ffb640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.122+0000 7fa57177e640 10 monclient: get_auth_request con 0x7fa55006e480 auth_method 0 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.122+0000 7fa561ffb640 10 client.4416.objecter ms_handle_connect 0x7fa55006e480 2024-03-29T11:00:26.112 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.132+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa55c04ff30 con 0x7fa56c1457f0 2024-03-29T11:00:26.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.132+0000 7fa561ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:26.113 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.132+0000 7fa561ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:26.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.461+0000 7fa5739e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:26.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.461+0000 7fa5739e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:26.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.461+0000 7fa5739e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:26.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.461+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa56c1d75e0 con 0x7fa56c1457f0 2024-03-29T11:00:26.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.463+0000 7fa561ffb640 1 -- 172.21.15.22:0/3832103533 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fa55c051e60 con 0x7fa56c1457f0 2024-03-29T11:00:26.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.463+0000 7fa561ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:26.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.463+0000 7fa561ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:26.443 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:26.443 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:26.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 10 librados: watch_flush enter 2024-03-29T11:00:26.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 10 librados: watch_flush exit 2024-03-29T11:00:26.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:00:26.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:00:26.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 >> v1:172.21.15.22:6816/380221644 conn(0x7fa55006e480 legacy=0x7fa550070940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:26.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 10 monclient: shutdown 2024-03-29T11:00:26.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:26.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.467+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 >> v1:172.21.15.22:6789/0 conn(0x7fa56c1457f0 legacy=0x7fa56c1350f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:26.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.469+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 shutdown_connections 2024-03-29T11:00:26.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.469+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 >> 172.21.15.22:0/3832103533 conn(0x7fa56c066a00 msgr2=0x7fa56c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:26.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.470+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 shutdown_connections 2024-03-29T11:00:26.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.470+0000 7fa5739e2640 1 -- 172.21.15.22:0/3832103533 wait complete. 2024-03-29T11:00:26.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:26.470+0000 7fa5739e2640 1 librados: shutdown 2024-03-29T11:00:26.472 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:33.278 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:33.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.447+0000 7f1ae71c1640 10 monclient: get_monmap_and_config 2024-03-29T11:00:33.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.447+0000 7f1ae71c1640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.427 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.447+0000 7f1ae71c1640 10 monclient: monmap: 2024-03-29T11:00:33.427 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:33.427 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:33.448259+0000 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:33.448259+0000 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:33.428 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.452+0000 7f1ae71c1640 1 Processor -- start 2024-03-29T11:00:33.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.452+0000 7f1ae71c1640 1 -- start start 2024-03-29T11:00:33.432 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.452+0000 7f1ae71c1640 10 monclient: init 2024-03-29T11:00:33.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.453+0000 7f1ae71c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.453+0000 7f1ae71c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:00:33.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 10 monclient(hunting): picked mon.noname-a con 0x7f1ae01422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:33.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae0142690 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:33.437 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae4f5d640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f1ae01422b0 0x7f1ae013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:37023/0 (socket says 172.21.15.22:37023) 2024-03-29T11:00:33.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 10 monclient(hunting): picked mon.noname-b con 0x7f1ae00b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:33.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae0142c80 con 0x7f1ae01422b0 2024-03-29T11:00:33.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae00aae40 con 0x7f1ae0142690 2024-03-29T11:00:33.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae013fbd0 con 0x7f1ae00b17d0 2024-03-29T11:00:33.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae4f5d640 1 -- 172.21.15.22:0/1328040465 learned_addr learned my addr 172.21.15.22:0/1328040465 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.457+0000 7f1ae71c1640 10 monclient(hunting): authenticate will time out at 1134.761908s 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821871951 0 0) 0x7f1ae0142c80 con 0x7f1ae01422b0 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac4003520 con 0x7f1ae01422b0 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1930198826 0 0) 0x7f1ae013fbd0 con 0x7f1ae00b17d0 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:00:33.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.458+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae0142c80 con 0x7f1ae00b17d0 2024-03-29T11:00:33.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.459+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3737767130 0 0) 0x7f1ae0142c80 con 0x7f1ae00b17d0 2024-03-29T11:00:33.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.459+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 >> v1:172.21.15.26:6789/0 conn(0x7f1ae0142690 legacy=0x7f1ae00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:33.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 >> v1:172.21.15.22:6789/0 conn(0x7f1ae01422b0 legacy=0x7f1ae013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.441 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 1 monclient: found mon.noname-b 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae0151940 con 0x7f1ae00b17d0 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1ad00015a0 con 0x7f1ae00b17d0 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:33.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: dump: 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1ae71c1640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1ae71c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ad0001700 con 0x7f1ae00b17d0 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 1 -- 172.21.15.22:0/1328040465 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1ad0001ad0 con 0x7f1ae00b17d0 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1adf7fe640 10 monclient: dump: 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.446 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1ae71c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:33.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.461+0000 7f1ae71c1640 4 set_mon_vals no callback set 2024-03-29T11:00:33.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.462+0000 7f1ae71c1640 10 monclient: shutdown 2024-03-29T11:00:33.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.462+0000 7f1ae71c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.462+0000 7f1ae71c1640 1 -- 172.21.15.22:0/1328040465 >> v1:172.21.15.22:6790/0 conn(0x7f1ae00b17d0 legacy=0x7f1ae00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.447 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.464+0000 7f1ae71c1640 1 -- 172.21.15.22:0/1328040465 shutdown_connections 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.464+0000 7f1ae71c1640 1 -- 172.21.15.22:0/1328040465 >> 172.21.15.22:0/1328040465 conn(0x7f1ae0066a00 msgr2=0x7f1ae0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.465+0000 7f1ae71c1640 1 -- 172.21.15.22:0/1328040465 shutdown_connections 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.465+0000 7f1ae71c1640 1 -- 172.21.15.22:0/1328040465 wait complete. 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.466+0000 7f1ae71c1640 10 monclient: build_initial_monmap 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.466+0000 7f1ae71c1640 10 monclient: monmap: 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:33.448 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.467+0000 7f1ae71c1640 1 librados: starting msgr at 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.467+0000 7f1ae71c1640 1 librados: starting objecter 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.467+0000 7f1ae71c1640 1 Processor -- start 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.467+0000 7f1ae71c1640 1 -- start start 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.467+0000 7f1ae71c1640 1 librados: setting wanted keys 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.467+0000 7f1ae71c1640 1 librados: calling monclient init 2024-03-29T11:00:33.449 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.467+0000 7f1ae71c1640 10 monclient: init 2024-03-29T11:00:33.450 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.468+0000 7f1ae71c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 10 monclient(hunting): picked mon.noname-c con 0x7f1ae00b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 10 monclient(hunting): picked mon.noname-b con 0x7f1ae01422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 10 monclient(hunting): picked mon.noname-a con 0x7f1ae0142690 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae012dae0 con 0x7f1ae0142690 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae012d860 con 0x7f1ae01422b0 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ae00aca80 con 0x7f1ae00b17d0 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae71c1640 10 monclient(hunting): authenticate will time out at 1134.773312s 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae4f5d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f1ae00b17d0 0x7f1ae012fba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:58244/0 (socket says 172.21.15.22:58244) 2024-03-29T11:00:33.451 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.469+0000 7f1ae4f5d640 1 -- 172.21.15.22:0/3578438198 learned_addr learned my addr 172.21.15.22:0/3578438198 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:33.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 10 client.?.objecter ms_handle_connect 0x7f1ae00b17d0 2024-03-29T11:00:33.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1905839401 0 0) 0x7f1ae00aca80 con 0x7f1ae00b17d0 2024-03-29T11:00:33.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:00:33.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.472+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac8004eb0 con 0x7f1ae00b17d0 2024-03-29T11:00:33.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.474+0000 7f1add7fa640 10 client.?.objecter ms_handle_connect 0x7f1ae01422b0 2024-03-29T11:00:33.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.474+0000 7f1add7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.474+0000 7f1add7fa640 10 client.?.objecter ms_handle_connect 0x7f1ae0142690 2024-03-29T11:00:33.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.474+0000 7f1add7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:33.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.474+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849918518 0 0) 0x7f1ae0151940 con 0x7f1ae0142690 2024-03-29T11:00:33.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.461 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae00aca80 con 0x7f1ae0142690 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3212886275 0 0) 0x7f1ac8004eb0 con 0x7f1ae00b17d0 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ae0151940 con 0x7f1ae00b17d0 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674950578 0 0) 0x7f1ae012d860 con 0x7f1ae01422b0 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ac8004eb0 con 0x7f1ae01422b0 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1acc001910 con 0x7f1ae00b17d0 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): dump: 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4025293505 0 0) 0x7f1ac8004eb0 con 0x7f1ae01422b0 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ae012d860 con 0x7f1ae01422b0 2024-03-29T11:00:33.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2573647750 0 0) 0x7f1ae00aca80 con 0x7f1ae0142690 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ac8004eb0 con 0x7f1ae0142690 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1ad8001a10 con 0x7f1ae01422b0 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): dump: 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:33.464 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1ad0000ed0 con 0x7f1ae0142690 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.475+0000 7f1add7fa640 10 monclient(hunting): dump: 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:33.465 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 805296095 0 0) 0x7f1ac8004eb0 con 0x7f1ae0142690 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 >> v1:172.21.15.22:6790/0 conn(0x7f1ae00b17d0 legacy=0x7f1ae012fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 >> v1:172.21.15.26:6789/0 conn(0x7f1ae01422b0 legacy=0x7f1ae012ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 1 monclient: found mon.a 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae00acd60 con 0x7f1ae0142690 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 10 monclient: _finish_auth 0 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 5 monclient: authenticate success, global_id 4422 2024-03-29T11:00:33.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 10 monclient: _renew_subs 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae01302b0 con 0x7f1ae0142690 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ad00035f0 con 0x7f1ae0142690 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 10 monclient: _renew_subs 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae0132130 con 0x7f1ae0142690 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.476+0000 7f1ae71c1640 1 librados: init done 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.478+0000 7f1add7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.479+0000 7f1adeffd640 4 set_mon_vals no callback set 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.479+0000 7f1ae71c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.480+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f1ad0003900 con 0x7f1ae0142690 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.480+0000 7f1add7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:33.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.480+0000 7f1add7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.480+0000 7f1add7fa640 10 monclient: dump: 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.480+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f1ad00198d0 con 0x7f1ae0142690 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.480+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f1ad0083650 con 0x7f1ae0142690 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1ae4f5d640 10 monclient: get_auth_request con 0x7f1ac806e1b0 auth_method 0 2024-03-29T11:00:33.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1ae71c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1ae71c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:33.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1abc005080 con 0x7f1ae0142690 2024-03-29T11:00:33.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1add7fa640 10 client.4422.objecter ms_dispatch 0x7f1ae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:33.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1add7fa640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:33.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1add7fa640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:33.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.481+0000 7f1add7fa640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-29T11:00:33.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.482+0000 7f1add7fa640 10 client.4422.objecter ms_handle_connect 0x7f1ac806e1b0 2024-03-29T11:00:33.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.490+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ad004efb0 con 0x7f1ae0142690 2024-03-29T11:00:33.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.490+0000 7f1add7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:33.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.490+0000 7f1add7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:33.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.828+0000 7f1ae71c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:33.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.828+0000 7f1ae71c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:33.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.828+0000 7f1ae71c1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:33.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.828+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1abc005330 con 0x7f1ae0142690 2024-03-29T11:00:33.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.829+0000 7f1add7fa640 1 -- 172.21.15.22:0/3578438198 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f1ad0050ee0 con 0x7f1ae0142690 2024-03-29T11:00:33.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.829+0000 7f1add7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:33.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.829+0000 7f1add7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:33.810 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:33.810 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:33.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.833+0000 7f1ae71c1640 10 librados: watch_flush enter 2024-03-29T11:00:33.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.833+0000 7f1ae71c1640 10 librados: watch_flush exit 2024-03-29T11:00:33.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.833+0000 7f1ae71c1640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-29T11:00:33.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.833+0000 7f1ae71c1640 10 client.4422.objecter successfully canceled tick 2024-03-29T11:00:33.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 >> v1:172.21.15.22:6816/380221644 conn(0x7f1ac806e1b0 legacy=0x7f1ac8070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 10 monclient: shutdown 2024-03-29T11:00:33.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:33.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 >> v1:172.21.15.22:6789/0 conn(0x7f1ae0142690 legacy=0x7f1ae012d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:33.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 shutdown_connections 2024-03-29T11:00:33.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 >> 172.21.15.22:0/3578438198 conn(0x7f1ae0066a00 msgr2=0x7f1ae013d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:33.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 shutdown_connections 2024-03-29T11:00:33.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 1 -- 172.21.15.22:0/3578438198 wait complete. 2024-03-29T11:00:33.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:33.834+0000 7f1ae71c1640 1 librados: shutdown 2024-03-29T11:00:33.831 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:40.639 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:40.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.810+0000 7fc88c263640 10 monclient: get_monmap_and_config 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.810+0000 7fc88c263640 10 monclient: build_initial_monmap 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.810+0000 7fc88c263640 10 monclient: monmap: 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:40.811085+0000 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:40.811085+0000 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:40.791 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:40.792 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:40.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.811+0000 7fc88c263640 1 Processor -- start 2024-03-29T11:00:40.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.812+0000 7fc88c263640 1 -- start start 2024-03-29T11:00:40.792 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.812+0000 7fc88c263640 10 monclient: init 2024-03-29T11:00:40.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:40.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:40.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 10 monclient(hunting): picked mon.noname-b con 0x7fc8841457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:40.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 10 monclient(hunting): picked mon.noname-a con 0x7fc88413f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:40.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 10 monclient(hunting): picked mon.noname-c con 0x7fc8840a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:40.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840aaef0 con 0x7fc88413f650 2024-03-29T11:00:40.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884145bd0 con 0x7fc8840a9d70 2024-03-29T11:00:40.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc884145eb0 con 0x7fc8841457f0 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc88c263640 10 monclient(hunting): authenticate will time out at 1142.117658s 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc8897fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fc88413f650 0x7fc8840b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:52248/0 (socket says 172.21.15.22:52248) 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.813+0000 7fc8897fe640 1 -- 172.21.15.22:0/140936735 learned_addr learned my addr 172.21.15.22:0/140936735 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914740896 0 0) 0x7fc8840aaef0 con 0x7fc88413f650 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:40.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc870003520 con 0x7fc88413f650 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1457211782 0 0) 0x7fc870003520 con 0x7fc88413f650 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 >> v1:172.21.15.22:6790/0 conn(0x7fc8841457f0 legacy=0x7fc88413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 >> v1:172.21.15.26:6789/0 conn(0x7fc8840a9d70 legacy=0x7fc88414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 monclient: found mon.noname-a 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:40.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884146340 con 0x7fc88413f650 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc88c263640 5 monclient: authenticate success, global_id 4425 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc88c263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc880001670 con 0x7fc88413f650 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:40.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc888ffd640 10 monclient: dump: 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:40.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.814+0000 7fc88c263640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc8800018d0 con 0x7fc88413f650 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc888ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc888ffd640 1 -- 172.21.15.22:0/140936735 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc880001670 con 0x7fc88413f650 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc888ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc888ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc888ffd640 10 monclient: dump: 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:40.801 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:40.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc88c263640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:40.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.815+0000 7fc88c263640 4 set_mon_vals no callback set 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.816+0000 7fc88c263640 10 monclient: shutdown 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.816+0000 7fc88c263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.816+0000 7fc88c263640 1 -- 172.21.15.22:0/140936735 >> v1:172.21.15.22:6789/0 conn(0x7fc88413f650 legacy=0x7fc8840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.816+0000 7fc88c263640 1 -- 172.21.15.22:0/140936735 shutdown_connections 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.816+0000 7fc88c263640 1 -- 172.21.15.22:0/140936735 >> 172.21.15.22:0/140936735 conn(0x7fc884066a00 msgr2=0x7fc884066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.816+0000 7fc88c263640 1 -- 172.21.15.22:0/140936735 shutdown_connections 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.816+0000 7fc88c263640 1 -- 172.21.15.22:0/140936735 wait complete. 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.818+0000 7fc88c263640 10 monclient: build_initial_monmap 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.818+0000 7fc88c263640 10 monclient: monmap: 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:40.806 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.818+0000 7fc88c263640 1 librados: starting msgr at 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.818+0000 7fc88c263640 1 librados: starting objecter 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.819+0000 7fc88c263640 1 Processor -- start 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.819+0000 7fc88c263640 1 -- start start 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.819+0000 7fc88c263640 1 librados: setting wanted keys 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.819+0000 7fc88c263640 1 librados: calling monclient init 2024-03-29T11:00:40.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.819+0000 7fc88c263640 10 monclient: init 2024-03-29T11:00:40.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.820+0000 7fc88c263640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:40.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.820+0000 7fc88c263640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:00:40.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.820+0000 7fc88c263640 10 monclient(hunting): picked mon.noname-b con 0x7fc8840a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:40.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc88c263640 10 monclient(hunting): picked mon.noname-a con 0x7fc88413f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:40.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc8897fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fc88413f650 0x7fc8840b0eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:57892/0 (socket says 172.21.15.22:57892) 2024-03-29T11:00:40.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc8897fe640 1 -- 172.21.15.22:0/554787632 learned_addr learned my addr 172.21.15.22:0/554787632 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:40.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc88c263640 10 monclient(hunting): picked mon.noname-c con 0x7fc8841457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc87a7fc640 10 client.?.objecter ms_handle_connect 0x7fc88413f650 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc87a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840af2e0 con 0x7fc88413f650 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8841d7ea0 con 0x7fc8840a9d70 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8840ae230 con 0x7fc8841457f0 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc88c263640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc88c263640 10 monclient(hunting): authenticate will time out at 1142.125813s 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc87a7fc640 10 client.?.objecter ms_handle_connect 0x7fc8840a9d70 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc87a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc87a7fc640 10 client.?.objecter ms_handle_connect 0x7fc8841457f0 2024-03-29T11:00:40.814 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.821+0000 7fc87a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714556863 0 0) 0x7fc8841d7ea0 con 0x7fc8840a9d70 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc868004eb0 con 0x7fc8840a9d70 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375183682 0 0) 0x7fc8840af2e0 con 0x7fc88413f650 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:00:40.815 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8841d7ea0 con 0x7fc88413f650 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1783172326 0 0) 0x7fc8840ae230 con 0x7fc8841457f0 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8840af2e0 con 0x7fc8841457f0 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786636040 0 0) 0x7fc868004eb0 con 0x7fc8840a9d70 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8840ae230 con 0x7fc8840a9d70 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc8740017c0 con 0x7fc8840a9d70 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.822+0000 7fc87a7fc640 10 monclient(hunting): dump: 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:40.816 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1478702174 0 0) 0x7fc8840ae230 con 0x7fc8840a9d70 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 >> v1:172.21.15.22:6790/0 conn(0x7fc8841457f0 legacy=0x7fc8840ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 >> v1:172.21.15.22:6789/0 conn(0x7fc88413f650 legacy=0x7fc8840b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 1 monclient: found mon.b 2024-03-29T11:00:40.817 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840ae3c0 con 0x7fc8840a9d70 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc87a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 5 monclient: authenticate success, global_id 4414 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 10 monclient: _renew_subs 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8840b15c0 con 0x7fc8840a9d70 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 10 monclient: _renew_subs 2024-03-29T11:00:40.818 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8841422e0 con 0x7fc8840a9d70 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.823+0000 7fc88c263640 1 librados: init done 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.824+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc874001cf0 con 0x7fc8840a9d70 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.824+0000 7fc87a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.824+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc8740037b0 con 0x7fc8840a9d70 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.824+0000 7fc87a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.824+0000 7fc87a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.824+0000 7fc87b7fe640 4 set_mon_vals no callback set 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.824+0000 7fc87a7fc640 10 monclient: dump: 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:40.819 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.825+0000 7fc88c263640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.825+0000 7fc88c263640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.825+0000 7fc88c263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.825+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc86c005080 con 0x7fc8840a9d70 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.827+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fc874003910 con 0x7fc8840a9d70 2024-03-29T11:00:40.820 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.828+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fc874083590 con 0x7fc8840a9d70 2024-03-29T11:00:40.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.828+0000 7fc87a7fc640 10 client.4414.objecter ms_dispatch 0x7fc884000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:40.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.828+0000 7fc87a7fc640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:40.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.828+0000 7fc87a7fc640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:40.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.828+0000 7fc87a7fc640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-29T11:00:40.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.828+0000 7fc8897fe640 10 monclient: get_auth_request con 0x7fc86806dde0 auth_method 0 2024-03-29T11:00:40.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.829+0000 7fc87a7fc640 10 client.4414.objecter ms_handle_connect 0x7fc86806dde0 2024-03-29T11:00:40.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.834+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc87404eca0 con 0x7fc8840a9d70 2024-03-29T11:00:40.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.834+0000 7fc87a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:40.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:40.834+0000 7fc87a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:41.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.172+0000 7fc88c263640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:41.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.172+0000 7fc88c263640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:41.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.172+0000 7fc88c263640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:41.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.172+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc86c005330 con 0x7fc8840a9d70 2024-03-29T11:00:41.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.173+0000 7fc87a7fc640 1 -- 172.21.15.22:0/554787632 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fc874003610 con 0x7fc8840a9d70 2024-03-29T11:00:41.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.173+0000 7fc87a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:41.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.173+0000 7fc87a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:41.160 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:41.161 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.178+0000 7fc88c263640 10 librados: watch_flush enter 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.178+0000 7fc88c263640 10 librados: watch_flush exit 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.178+0000 7fc88c263640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.178+0000 7fc88c263640 10 client.4414.objecter successfully canceled tick 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.179+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 >> v1:172.21.15.22:6816/380221644 conn(0x7fc86806dde0 legacy=0x7fc8680702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.179+0000 7fc88c263640 10 monclient: shutdown 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.179+0000 7fc88c263640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.179+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 >> v1:172.21.15.26:6789/0 conn(0x7fc8840a9d70 legacy=0x7fc8841d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:41.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.180+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 shutdown_connections 2024-03-29T11:00:41.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.180+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 >> 172.21.15.22:0/554787632 conn(0x7fc884066a00 msgr2=0x7fc884066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:41.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.180+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 shutdown_connections 2024-03-29T11:00:41.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.180+0000 7fc88c263640 1 -- 172.21.15.22:0/554787632 wait complete. 2024-03-29T11:00:41.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:41.181+0000 7fc88c263640 1 librados: shutdown 2024-03-29T11:00:41.179 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:47.986 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:48.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.161+0000 7fcf83ec1640 10 monclient: get_monmap_and_config 2024-03-29T11:00:48.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.161+0000 7fcf83ec1640 10 monclient: build_initial_monmap 2024-03-29T11:00:48.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.161+0000 7fcf83ec1640 10 monclient: monmap: 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:48.162265+0000 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:48.162265+0000 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:48.142 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:48.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.162+0000 7fcf83ec1640 1 Processor -- start 2024-03-29T11:00:48.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.162+0000 7fcf83ec1640 1 -- start start 2024-03-29T11:00:48.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.162+0000 7fcf83ec1640 10 monclient: init 2024-03-29T11:00:48.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 10 monclient(hunting): picked mon.noname-c con 0x7fcf7c0b2080 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 10 monclient(hunting): picked mon.noname-b con 0x7fcf7c0b65c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 10 monclient(hunting): picked mon.noname-a con 0x7fcf7c0b6aa0 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c0b43b0 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c0b3440 con 0x7fcf7c0b2080 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c0b4600 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.163+0000 7fcf83ec1640 10 monclient(hunting): authenticate will time out at 1149.467920s 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf8245e640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fcf7c0b6aa0 0x7fcf7c151c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:33590/0 (socket says 172.21.15.22:33590) 2024-03-29T11:00:48.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf8245e640 1 -- 172.21.15.22:0/836704347 learned_addr learned my addr 172.21.15.22:0/836704347 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:48.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879737812 0 0) 0x7fcf7c0b43b0 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf80c5b640 10 monclient(hunting): _negotiate 2024-03-29T11:00:48.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf80c5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf80c5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf80c5b640 10 monclient(hunting): my global_id is 4431 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf68003520 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.164+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653831880 0 0) 0x7fcf7c0b4600 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient(hunting): _negotiate 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7c0b43b0 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2349613504 0 0) 0x7fcf68003520 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 >> v1:172.21.15.22:6790/0 conn(0x7fcf7c0b65c0 legacy=0x7fcf7c1437a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 >> v1:172.21.15.26:6789/0 conn(0x7fcf7c0b2080 legacy=0x7fcf7c14a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: _finish_hunting 0 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 monclient: found mon.noname-a 2024-03-29T11:00:48.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c14a720 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: _finish_auth 0 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: _check_auth_tickets 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fcf74001a10 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf83ec1640 5 monclient: authenticate success, global_id 4431 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf83ec1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: dump: 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:48.149 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf74001c30 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 1 -- 172.21.15.22:0/836704347 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fcf74001fc0 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf80c5b640 10 monclient: dump: 2024-03-29T11:00:48.150 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf83ec1640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.165+0000 7fcf83ec1640 4 set_mon_vals no callback set 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.167+0000 7fcf83ec1640 10 monclient: shutdown 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.167+0000 7fcf83ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.167+0000 7fcf83ec1640 1 -- 172.21.15.22:0/836704347 >> v1:172.21.15.22:6789/0 conn(0x7fcf7c0b6aa0 legacy=0x7fcf7c151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.167+0000 7fcf83ec1640 1 -- 172.21.15.22:0/836704347 shutdown_connections 2024-03-29T11:00:48.151 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.167+0000 7fcf83ec1640 1 -- 172.21.15.22:0/836704347 >> 172.21.15.22:0/836704347 conn(0x7fcf7c0bc0a0 msgr2=0x7fcf7c0bc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:48.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.167+0000 7fcf83ec1640 1 -- 172.21.15.22:0/836704347 shutdown_connections 2024-03-29T11:00:48.152 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.168+0000 7fcf83ec1640 1 -- 172.21.15.22:0/836704347 wait complete. 2024-03-29T11:00:48.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.169+0000 7fcf83ec1640 10 monclient: build_initial_monmap 2024-03-29T11:00:48.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.169+0000 7fcf83ec1640 10 monclient: monmap: 2024-03-29T11:00:48.153 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:48.153 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:48.153 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:48.153 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.170+0000 7fcf83ec1640 1 librados: starting msgr at 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.170+0000 7fcf83ec1640 1 librados: starting objecter 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.170+0000 7fcf83ec1640 1 Processor -- start 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.170+0000 7fcf83ec1640 1 -- start start 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.170+0000 7fcf83ec1640 1 librados: setting wanted keys 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.170+0000 7fcf83ec1640 1 librados: calling monclient init 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.170+0000 7fcf83ec1640 10 monclient: init 2024-03-29T11:00:48.154 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 10 monclient(hunting): picked mon.noname-c con 0x7fcf7c0b2080 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 10 monclient(hunting): picked mon.noname-a con 0x7fcf7c0b65c0 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf8145c640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fcf7c0b65c0 0x7fcf7c145890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34548/0 (socket says 172.21.15.22:34548) 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 10 monclient(hunting): picked mon.noname-b con 0x7fcf7c0b6aa0 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c13a860 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf8145c640 1 -- 172.21.15.22:0/1877009580 learned_addr learned my addr 172.21.15.22:0/1877009580 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c13bc50 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf7c13bf30 con 0x7fcf7c0b2080 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf83ec1640 10 monclient(hunting): authenticate will time out at 1149.475672s 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf727fc640 10 client.?.objecter ms_handle_connect 0x7fcf7c0b65c0 2024-03-29T11:00:48.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf727fc640 10 client.?.objecter ms_handle_connect 0x7fcf7c0b6aa0 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf727fc640 10 client.?.objecter ms_handle_connect 0x7fcf7c0b2080 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.171+0000 7fcf727fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246916745 0 0) 0x7fcf7c13a860 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf60004eb0 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1761937274 0 0) 0x7fcf7c13bc50 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.156 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7c13a860 con 0x7fcf7c0b6aa0 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3180550552 0 0) 0x7fcf7c13bf30 con 0x7fcf7c0b2080 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _negotiate 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.172+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7c13bc50 con 0x7fcf7c0b2080 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1302332192 0 0) 0x7fcf7c13bc50 con 0x7fcf7c0b2080 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf7c13bf30 con 0x7fcf7c0b2080 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fcf78001780 con 0x7fcf7c0b2080 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:48.157 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient(hunting): dump: 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 433879217 0 0) 0x7fcf60004eb0 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.158 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf7c13bc50 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fcf6c001990 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient(hunting): dump: 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:48.159 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 983188466 0 0) 0x7fcf7c13bc50 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 >> v1:172.21.15.22:6790/0 conn(0x7fcf7c0b2080 legacy=0x7fcf7c13c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 >> v1:172.21.15.26:6789/0 conn(0x7fcf7c0b6aa0 legacy=0x7fcf7c1e6400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient: _finish_hunting 0 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 monclient: found mon.a 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf7c13c210 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.173+0000 7fcf727fc640 10 monclient: _finish_auth 0 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 10 monclient: _check_auth_tickets 2024-03-29T11:00:48.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:48.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 5 monclient: authenticate success, global_id 4434 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 10 monclient: _renew_subs 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf7c13b160 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 10 monclient: _renew_subs 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf7c13a260 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf83ec1640 1 librados: init done 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf6c003610 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf73fff640 4 set_mon_vals no callback set 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fcf6c003a20 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.174+0000 7fcf727fc640 10 monclient: dump: 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.175+0000 7fcf83ec1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.175+0000 7fcf83ec1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.175+0000 7fcf83ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:48.163 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.175+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf64005080 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.176+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fcf6c003b80 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.177+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fcf6c084a70 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.177+0000 7fcf727fc640 10 client.4434.objecter ms_dispatch 0x7fcf7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.177+0000 7fcf727fc640 3 client.4434.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.177+0000 7fcf727fc640 3 client.4434.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.177+0000 7fcf727fc640 20 client.4434.objecter dump_active .. 0 homeless 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.177+0000 7fcf81c5d640 10 monclient: get_auth_request con 0x7fcf6006ddb0 auth_method 0 2024-03-29T11:00:48.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.177+0000 7fcf727fc640 10 client.4434.objecter ms_handle_connect 0x7fcf6006ddb0 2024-03-29T11:00:48.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.186+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf6c050350 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.186+0000 7fcf727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:48.166 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.186+0000 7fcf727fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:48.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.565+0000 7fcf83ec1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:48.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.565+0000 7fcf83ec1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:48.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.565+0000 7fcf83ec1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:00:48.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.565+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf64005330 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.566+0000 7fcf727fc640 1 -- 172.21.15.22:0/1877009580 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fcf6c052280 con 0x7fcf7c0b65c0 2024-03-29T11:00:48.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.566+0000 7fcf727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:48.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.566+0000 7fcf727fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:48.547 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:48.547 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 10 librados: watch_flush enter 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 10 librados: watch_flush exit 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 20 client.4434.objecter shutdown clearing up homeless session... 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 10 client.4434.objecter successfully canceled tick 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 >> v1:172.21.15.22:6816/380221644 conn(0x7fcf6006ddb0 legacy=0x7fcf60070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 10 monclient: shutdown 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:48.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 >> v1:172.21.15.22:6789/0 conn(0x7fcf7c0b65c0 legacy=0x7fcf7c145890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:48.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 shutdown_connections 2024-03-29T11:00:48.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 >> 172.21.15.22:0/1877009580 conn(0x7fcf7c0bc0a0 msgr2=0x7fcf7c0b6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:48.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 shutdown_connections 2024-03-29T11:00:48.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.570+0000 7fcf83ec1640 1 -- 172.21.15.22:0/1877009580 wait complete. 2024-03-29T11:00:48.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:48.571+0000 7fcf83ec1640 1 librados: shutdown 2024-03-29T11:00:48.564 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:00:55.371 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:00:55.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.547+0000 7f94ae7f4640 10 monclient: get_monmap_and_config 2024-03-29T11:00:55.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.547+0000 7f94ae7f4640 10 monclient: build_initial_monmap 2024-03-29T11:00:55.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.547+0000 7f94ae7f4640 10 monclient: monmap: 2024-03-29T11:00:55.528 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:00:55.548398+0000 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:00:55.548398+0000 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:00:55.529 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:55.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.550+0000 7f94ae7f4640 1 Processor -- start 2024-03-29T11:00:55.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.551+0000 7f94ae7f4640 1 -- start start 2024-03-29T11:00:55.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.551+0000 7f94ae7f4640 10 monclient: init 2024-03-29T11:00:55.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.551+0000 7f94ae7f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:55.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.551+0000 7f94ae7f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:00:55.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.551+0000 7f94ae7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f94a80a9ba0 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:55.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.551+0000 7f94ae7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f94a813b5a0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:55.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.555+0000 7f94a7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f94a80a9ba0 0x7f94a813ae90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:43525/0 (socket says 172.21.15.22:43525) 2024-03-29T11:00:55.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94ae7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f94a80ab550 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:55.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94ae7f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a80ac430 con 0x7f94a80ab550 2024-03-29T11:00:55.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94ae7f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a80ac680 con 0x7f94a80a9ba0 2024-03-29T11:00:55.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94ae7f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a80ac960 con 0x7f94a813b5a0 2024-03-29T11:00:55.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94ae7f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:55.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94ae7f4640 10 monclient(hunting): authenticate will time out at 1156.861471s 2024-03-29T11:00:55.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94a77fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f94a813b5a0 0x7f94a80aae40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39538/0 (socket says 172.21.15.22:39538) 2024-03-29T11:00:55.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.557+0000 7f94a77fe640 1 -- 172.21.15.22:0/638963315 learned_addr learned my addr 172.21.15.22:0/638963315 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.558+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888050569 0 0) 0x7f94a80ac680 con 0x7f94a80a9ba0 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.558+0000 7f94a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.558+0000 7f94a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.558+0000 7f94a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.558+0000 7f94a6ffd640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.558+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9490003520 con 0x7f94a80a9ba0 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 471727311 0 0) 0x7f94a80ac960 con 0x7f94a813b5a0 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient(hunting): my global_id is 4544 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a80ac680 con 0x7f94a813b5a0 2024-03-29T11:00:55.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442663204 0 0) 0x7f9490003520 con 0x7f94a80a9ba0 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 >> v1:172.21.15.22:6790/0 conn(0x7f94a813b5a0 legacy=0x7f94a80aae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 >> v1:172.21.15.22:6789/0 conn(0x7f94a80ab550 legacy=0x7f94a81498d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a813baf0 con 0x7f94a80a9ba0 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:00:55.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.559+0000 7f94a6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:00:55.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.561+0000 7f94a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:55.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.561+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f949c001690 con 0x7f94a80a9ba0 2024-03-29T11:00:55.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.561+0000 7f94a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:55.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.561+0000 7f94a6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:00:55.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.561+0000 7f94a6ffd640 10 monclient: dump: 2024-03-29T11:00:55.543 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:55.543 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.562+0000 7f94ae7f4640 5 monclient: authenticate success, global_id 4423 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.562+0000 7f94ae7f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f949c0018b0 con 0x7f94a80a9ba0 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94a6ffd640 1 -- 172.21.15.22:0/638963315 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f949c001c40 con 0x7f94a80a9ba0 2024-03-29T11:00:55.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94a6ffd640 10 monclient: dump: 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94ae7f4640 10 monclient: get_monmap_and_config success 2024-03-29T11:00:55.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.563+0000 7f94ae7f4640 4 set_mon_vals no callback set 2024-03-29T11:00:55.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.564+0000 7f94ae7f4640 10 monclient: shutdown 2024-03-29T11:00:55.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.564+0000 7f94ae7f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.564+0000 7f94ae7f4640 1 -- 172.21.15.22:0/638963315 >> v1:172.21.15.26:6789/0 conn(0x7f94a80a9ba0 legacy=0x7f94a813ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.564+0000 7f94ae7f4640 1 -- 172.21.15.22:0/638963315 shutdown_connections 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.564+0000 7f94ae7f4640 1 -- 172.21.15.22:0/638963315 >> 172.21.15.22:0/638963315 conn(0x7f94a80b3a00 msgr2=0x7f94a80b3de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.564+0000 7f94ae7f4640 1 -- 172.21.15.22:0/638963315 shutdown_connections 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.564+0000 7f94ae7f4640 1 -- 172.21.15.22:0/638963315 wait complete. 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.565+0000 7f94ae7f4640 10 monclient: build_initial_monmap 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.565+0000 7f94ae7f4640 10 monclient: monmap: 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:00:55.548 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.566+0000 7f94ae7f4640 1 librados: starting msgr at 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.566+0000 7f94ae7f4640 1 librados: starting objecter 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.566+0000 7f94ae7f4640 1 Processor -- start 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.567+0000 7f94ae7f4640 1 -- start start 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.567+0000 7f94ae7f4640 1 librados: setting wanted keys 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.567+0000 7f94ae7f4640 1 librados: calling monclient init 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.567+0000 7f94ae7f4640 10 monclient: init 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.568+0000 7f94ae7f4640 10 monclient: _reopen_session rank -1 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.568+0000 7f94ae7f4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.568+0000 7f94ae7f4640 10 monclient(hunting): picked mon.noname-c con 0x7f94a80a9ba0 addr v1:172.21.15.22:6790/0 2024-03-29T11:00:55.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.568+0000 7f94ae7f4640 10 monclient(hunting): picked mon.noname-b con 0x7f94a80ab550 addr v1:172.21.15.26:6789/0 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94a7fff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f94a80a9ba0 0x7f94a81d63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40036/0 (socket says 172.21.15.22:40036) 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94ae7f4640 10 monclient(hunting): picked mon.noname-a con 0x7f94a813b5a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94ae7f4640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a806da60 con 0x7f94a813b5a0 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94ae7f4640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a81d6c70 con 0x7f94a80ab550 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94ae7f4640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a81422e0 con 0x7f94a80a9ba0 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94ae7f4640 10 monclient(hunting): _renew_subs 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94ae7f4640 10 monclient(hunting): authenticate will time out at 1156.877687s 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94a7fff640 1 -- 172.21.15.22:0/1505243466 learned_addr learned my addr 172.21.15.22:0/1505243466 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:00:55.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94a77fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f94a80ab550 0x7f94a813ff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:43653/0 (socket says 172.21.15.22:43653) 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94a4ff9640 10 client.?.objecter ms_handle_connect 0x7f94a80a9ba0 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.573+0000 7f94a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 10 client.?.objecter ms_handle_connect 0x7f94a80ab550 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3619102933 0 0) 0x7f94a81d6c70 con 0x7f94a80ab550 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.574+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9494004eb0 con 0x7f94a80ab550 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619022259 0 0) 0x7f9494004eb0 con 0x7f94a80ab550 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94a81d6c70 con 0x7f94a80ab550 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9498001850 con 0x7f94a80ab550 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:55.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 10 monclient(hunting): dump: 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3906521286 0 0) 0x7f94a81d6c70 con 0x7f94a80ab550 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 >> v1:172.21.15.22:6790/0 conn(0x7f94a80a9ba0 legacy=0x7f94a81d63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:55.557 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 >> v1:172.21.15.22:6789/0 conn(0x7f94a813b5a0 legacy=0x7f94a8140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:55.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:00:55.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 monclient: found mon.b 2024-03-29T11:00:55.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:00:55.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:55.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a81367d0 con 0x7f94a80ab550 2024-03-29T11:00:55.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.575+0000 7f94a4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94a4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94a4ff9640 10 client.?.objecter ms_handle_connect 0x7f94a813b5a0 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 10 monclient: _renew_subs 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:55.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 1 -- 172.21.15.22:0/1505243466 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a8136a80 con 0x7f94a80ab550 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 10 monclient: _renew_subs 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 1 -- 172.21.15.22:0/1505243466 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a8072130 con 0x7f94a80ab550 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.576+0000 7f94ae7f4640 1 librados: init done 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.577+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f94980047e0 con 0x7f94a80ab550 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.577+0000 7f94a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.577+0000 7f94a67fc640 4 set_mon_vals no callback set 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f9498004af0 con 0x7f94a80ab550 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f94a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f94a4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f94a4ff9640 10 monclient: dump: 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:00:55.561 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:00:55.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f948e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:55.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f948e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:55.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f948e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:55.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.578+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a804f730 con 0x7f94a80ab550 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.580+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f949801aac0 con 0x7f94a80ab550 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.581+0000 7f94a7fff640 10 monclient: get_auth_request con 0x7f949406db80 auth_method 0 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.581+0000 7f94a4ff9640 10 client.4426.objecter ms_handle_connect 0x7f949406db80 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.581+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f9498054890 con 0x7f94a80ab550 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.581+0000 7f94a4ff9640 10 client.4426.objecter ms_dispatch 0x7f94a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.581+0000 7f94a4ff9640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.581+0000 7f94a4ff9640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:00:55.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.581+0000 7f94a4ff9640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-29T11:00:55.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.588+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9498050b30 con 0x7f94a80ab550 2024-03-29T11:00:55.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.588+0000 7f94a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:00:55.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.588+0000 7f94a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:00:55.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.972+0000 7f948e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:55.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.972+0000 7f948e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:55.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.972+0000 7f948e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:00:55.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.972+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94a8074e10 con 0x7f94a80ab550 2024-03-29T11:00:55.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.972+0000 7f94a4ff9640 1 -- 172.21.15.22:0/1505243466 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f9498050b30 con 0x7f94a80ab550 2024-03-29T11:00:55.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.972+0000 7f94a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:00:55.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.972+0000 7f94a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:00:55.953 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:00:55.953 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.975+0000 7f948e7fc640 10 librados: watch_flush enter 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 10 librados: watch_flush exit 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 10 client.4426.objecter successfully canceled tick 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 >> v1:172.21.15.22:6816/380221644 conn(0x7f949406db80 legacy=0x7f9494070040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 10 monclient: shutdown 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 >> v1:172.21.15.26:6789/0 conn(0x7f94a80ab550 legacy=0x7f94a813ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:00:55.956 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 shutdown_connections 2024-03-29T11:00:55.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 >> 172.21.15.22:0/1505243466 conn(0x7f94a80b3a00 msgr2=0x7f94a8069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:00:55.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 shutdown_connections 2024-03-29T11:00:55.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 1 -- 172.21.15.22:0/1505243466 wait complete. 2024-03-29T11:00:55.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:00:55.976+0000 7f948e7fc640 1 librados: shutdown 2024-03-29T11:00:55.971 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:02.777 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:02.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.951+0000 7f58c192b640 10 monclient: get_monmap_and_config 2024-03-29T11:01:02.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.951+0000 7f58c192b640 10 monclient: build_initial_monmap 2024-03-29T11:01:02.932 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.951+0000 7f58c192b640 10 monclient: monmap: 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:02.953585+0000 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:02.953585+0000 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:02.933 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.953+0000 7f58c192b640 1 Processor -- start 2024-03-29T11:01:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.953+0000 7f58c192b640 1 -- start start 2024-03-29T11:01:02.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.953+0000 7f58c192b640 10 monclient: init 2024-03-29T11:01:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:01:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 10 monclient(hunting): picked mon.noname-c con 0x7f58bc1422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:02.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc0b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc142c80 con 0x7f58bc0b17d0 2024-03-29T11:01:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0aae40 con 0x7f58bc1422b0 2024-03-29T11:01:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc13fbd0 con 0x7f58bc142690 2024-03-29T11:01:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.954+0000 7f58c192b640 10 monclient(hunting): authenticate will time out at 1164.259955s 2024-03-29T11:01:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.955+0000 7f58ba7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f58bc142690 0x7f58bc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42272/0 (socket says 172.21.15.22:42272) 2024-03-29T11:01:02.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.955+0000 7f58ba7fc640 1 -- 172.21.15.22:0/2496253440 learned_addr learned my addr 172.21.15.22:0/2496253440 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.956+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539795950 0 0) 0x7f58bc13fbd0 con 0x7f58bc142690 2024-03-29T11:01:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.956+0000 7f58b9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.956+0000 7f58b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.956+0000 7f58b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.956+0000 7f58b9ffb640 10 monclient(hunting): my global_id is 4550 2024-03-29T11:01:02.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.956+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a4003520 con 0x7f58bc142690 2024-03-29T11:01:02.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.957+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943863783 0 0) 0x7f58a4003520 con 0x7f58bc142690 2024-03-29T11:01:02.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.957+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 >> v1:172.21.15.26:6789/0 conn(0x7f58bc1422b0 legacy=0x7f58bc13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 >> v1:172.21.15.22:6789/0 conn(0x7f58bc0b17d0 legacy=0x7f58bc0b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc151940 con 0x7f58bc142690 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:02.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f58b0001690 con 0x7f58bc142690 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.959+0000 7f58b9ffb640 10 monclient: dump: 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:02.943 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b00018b0 con 0x7f58bc142690 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58b9ffb640 1 -- 172.21.15.22:0/2496253440 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f58b0001c40 con 0x7f58bc142690 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58b9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58b9ffb640 10 monclient: dump: 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:02.944 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58c192b640 5 monclient: authenticate success, global_id 4550 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58c192b640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.960+0000 7f58c192b640 4 set_mon_vals no callback set 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.961+0000 7f58c192b640 10 monclient: shutdown 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.961+0000 7f58c192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.961+0000 7f58c192b640 1 -- 172.21.15.22:0/2496253440 >> v1:172.21.15.22:6790/0 conn(0x7f58bc142690 legacy=0x7f58bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.961+0000 7f58c192b640 1 -- 172.21.15.22:0/2496253440 shutdown_connections 2024-03-29T11:01:02.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.961+0000 7f58c192b640 1 -- 172.21.15.22:0/2496253440 >> 172.21.15.22:0/2496253440 conn(0x7f58bc066a00 msgr2=0x7f58bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.961+0000 7f58c192b640 1 -- 172.21.15.22:0/2496253440 shutdown_connections 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.961+0000 7f58c192b640 1 -- 172.21.15.22:0/2496253440 wait complete. 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.962+0000 7f58c192b640 10 monclient: build_initial_monmap 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.962+0000 7f58c192b640 10 monclient: monmap: 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:02.946 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:02.947 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:02.947 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:02.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.963+0000 7f58c192b640 1 librados: starting msgr at 2024-03-29T11:01:02.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.963+0000 7f58c192b640 1 librados: starting objecter 2024-03-29T11:01:02.947 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.963+0000 7f58c192b640 1 Processor -- start 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.963+0000 7f58c192b640 1 -- start start 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.963+0000 7f58c192b640 1 librados: setting wanted keys 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.963+0000 7f58c192b640 1 librados: calling monclient init 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.963+0000 7f58c192b640 10 monclient: init 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 10 monclient(hunting): picked mon.noname-a con 0x7f58bc0b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:02.948 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 10 monclient(hunting): picked mon.noname-b con 0x7f58bc1422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 10 monclient(hunting): picked mon.noname-c con 0x7f58bc142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0b0df0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc0709c0 con 0x7f58bc1422b0 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58bc070ca0 con 0x7f58bc142690 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.964+0000 7f58c192b640 10 monclient(hunting): authenticate will time out at 1164.269792s 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.968+0000 7f58bb7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f58bc142690 0x7f58bc06f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:42790/0 (socket says 172.21.15.22:42790) 2024-03-29T11:01:02.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.968+0000 7f58bb7fe640 1 -- 172.21.15.22:0/3221228698 learned_addr learned my addr 172.21.15.22:0/3221228698 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:02.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.969+0000 7f58a3fff640 10 client.?.objecter ms_handle_connect 0x7f58bc142690 2024-03-29T11:01:02.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.969+0000 7f58a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:02.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.969+0000 7f58a3fff640 10 client.?.objecter ms_handle_connect 0x7f58bc0b17d0 2024-03-29T11:01:02.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.969+0000 7f58a3fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:02.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2620990185 0 0) 0x7f58bc0b0df0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a8004eb0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520175121 0 0) 0x7f58bc070ca0 con 0x7f58bc142690 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:01:02.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58bc0b0df0 con 0x7f58bc142690 2024-03-29T11:01:02.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2033574695 0 0) 0x7f58a8004eb0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58bc070ca0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f58ac001870 con 0x7f58bc0b17d0 2024-03-29T11:01:02.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): dump: 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:02.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2525651758 0 0) 0x7f58bc070ca0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 >> v1:172.21.15.22:6790/0 conn(0x7f58bc142690 legacy=0x7f58bc06f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.970+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 >> v1:172.21.15.26:6789/0 conn(0x7f58bc1422b0 legacy=0x7f58bc06ec90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 1 monclient: found mon.a 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58bc070f80 con 0x7f58bc0b17d0 2024-03-29T11:01:02.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 5 monclient: authenticate success, global_id 4437 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 10 monclient: _renew_subs 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58bc0b0840 con 0x7f58bc0b17d0 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 10 client.4437.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 10 monclient: _renew_subs 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58bc071710 con 0x7f58bc0b17d0 2024-03-29T11:01:02.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.971+0000 7f58c192b640 1 librados: init done 2024-03-29T11:01:02.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.973+0000 7f58c192b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:02.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.973+0000 7f58c192b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:02.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.973+0000 7f58c192b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:02.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.973+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58bc075f10 con 0x7f58bc0b17d0 2024-03-29T11:01:02.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.975+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58ac0019d0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.975+0000 7f58a3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:02.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.975+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f58ac003830 con 0x7f58bc0b17d0 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.975+0000 7f58a3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.975+0000 7f58a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.975+0000 7f58a3fff640 10 monclient: dump: 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:02.958 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:02.959 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:02.959 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:02.959 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:02.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.975+0000 7f58b97fa640 4 set_mon_vals no callback set 2024-03-29T11:01:02.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.976+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f58ac019800 con 0x7f58bc0b17d0 2024-03-29T11:01:02.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.977+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f58ac0837d0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.977+0000 7f58a3fff640 10 client.4437.objecter ms_dispatch 0x7f58bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:02.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.977+0000 7f58a3fff640 3 client.4437.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:02.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.977+0000 7f58a3fff640 3 client.4437.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:02.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.977+0000 7f58a3fff640 20 client.4437.objecter dump_active .. 0 homeless 2024-03-29T11:01:02.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.977+0000 7f58ba7fc640 10 monclient: get_auth_request con 0x7f58a806dcb0 auth_method 0 2024-03-29T11:01:02.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.978+0000 7f58a3fff640 10 client.4437.objecter ms_handle_connect 0x7f58a806dcb0 2024-03-29T11:01:02.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.983+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58ac04f1f0 con 0x7f58bc0b17d0 2024-03-29T11:01:02.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.983+0000 7f58a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:02.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:02.983+0000 7f58a3fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:03.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.291+0000 7f58c192b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:03.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.291+0000 7f58c192b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:03.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.291+0000 7f58c192b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:03.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.291+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58bc0b2270 con 0x7f58bc0b17d0 2024-03-29T11:01:03.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.292+0000 7f58a3fff640 1 -- 172.21.15.22:0/3221228698 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f58ac051120 con 0x7f58bc0b17d0 2024-03-29T11:01:03.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.292+0000 7f58a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:03.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.292+0000 7f58a3fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:03.274 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:03.274 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:03.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 10 librados: watch_flush enter 2024-03-29T11:01:03.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 10 librados: watch_flush exit 2024-03-29T11:01:03.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 20 client.4437.objecter shutdown clearing up homeless session... 2024-03-29T11:01:03.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 10 client.4437.objecter successfully canceled tick 2024-03-29T11:01:03.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 >> v1:172.21.15.22:6816/380221644 conn(0x7f58a806dcb0 legacy=0x7f58a8070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:03.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 10 monclient: shutdown 2024-03-29T11:01:03.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:03.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 >> v1:172.21.15.22:6789/0 conn(0x7f58bc0b17d0 legacy=0x7f58bc13bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:03.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 shutdown_connections 2024-03-29T11:01:03.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.296+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 >> 172.21.15.22:0/3221228698 conn(0x7f58bc066a00 msgr2=0x7f58bc0b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:03.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.298+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 shutdown_connections 2024-03-29T11:01:03.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.298+0000 7f58c192b640 1 -- 172.21.15.22:0/3221228698 wait complete. 2024-03-29T11:01:03.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:03.298+0000 7f58c192b640 1 librados: shutdown 2024-03-29T11:01:03.296 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:10.102 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.254+0000 7fe44beaf640 10 monclient: get_monmap_and_config 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.254+0000 7fe44beaf640 10 monclient: build_initial_monmap 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.254+0000 7fe44beaf640 10 monclient: monmap: 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:10.256401+0000 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:10.256401+0000 2024-03-29T11:01:10.235 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:10.236 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:10.236 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:10.236 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:10.236 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:10.236 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:10.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.255+0000 7fe44beaf640 1 Processor -- start 2024-03-29T11:01:10.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.255+0000 7fe44beaf640 1 -- start start 2024-03-29T11:01:10.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.255+0000 7fe44beaf640 10 monclient: init 2024-03-29T11:01:10.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.256+0000 7fe44beaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:10.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.256+0000 7fe44beaf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:10.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.256+0000 7fe44beaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe4441422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:10.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.256+0000 7fe44beaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe444142690 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe449c4b640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fe4441422b0 0x7fe44413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:58620/0 (socket says 172.21.15.22:58620) 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe44a44c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe4440b17d0 0x7fe4440b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:56564/0 (socket says 172.21.15.22:56564) 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe449c4b640 1 -- 172.21.15.22:0/375674690 learned_addr learned my addr 172.21.15.22:0/375674690 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe44beaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe4440b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe444142c80 con 0x7fe444142690 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440aae40 con 0x7fe4441422b0 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe44413fbd0 con 0x7fe4440b17d0 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe44beaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1278460082 0 0) 0x7fe4440aae40 con 0x7fe4441422b0 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.257+0000 7fe44beaf640 10 monclient(hunting): authenticate will time out at 1171.562989s 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): _negotiate 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:10.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe430003520 con 0x7fe4441422b0 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1256571548 0 0) 0x7fe44413fbd0 con 0x7fe4440b17d0 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): _negotiate 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.258+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4440aae40 con 0x7fe4440b17d0 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.259+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 555108379 0 0) 0x7fe4440aae40 con 0x7fe4440b17d0 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.259+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 >> v1:172.21.15.26:6789/0 conn(0x7fe4441422b0 legacy=0x7fe44413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.259+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 >> v1:172.21.15.22:6789/0 conn(0x7fe444142690 legacy=0x7fe4440b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.259+0000 7fe448c49640 10 monclient: _finish_hunting 0 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.259+0000 7fe448c49640 1 monclient: found mon.noname-b 2024-03-29T11:01:10.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.259+0000 7fe448c49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe444150930 con 0x7fe4440b17d0 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 10 monclient: _finish_auth 0 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 10 monclient: _check_auth_tickets 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe43c0019f0 con 0x7fe4440b17d0 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe448c49640 10 monclient: dump: 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.243 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:10.244 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:10.244 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:10.244 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:10.244 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:10.244 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:10.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe44beaf640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:01:10.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.260+0000 7fe44beaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe43c001b50 con 0x7fe4440b17d0 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe448c49640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe448c49640 1 -- 172.21.15.22:0/375674690 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe43c001f20 con 0x7fe4440b17d0 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe448c49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe448c49640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe448c49640 10 monclient: dump: 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:10.245 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe44beaf640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.262+0000 7fe44beaf640 4 set_mon_vals no callback set 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.263+0000 7fe44beaf640 10 monclient: shutdown 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.263+0000 7fe44beaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.263+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 >> v1:172.21.15.22:6790/0 conn(0x7fe4440b17d0 legacy=0x7fe4440b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.264+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 shutdown_connections 2024-03-29T11:01:10.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.264+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 >> 172.21.15.22:0/375674690 conn(0x7fe444066a00 msgr2=0x7fe444066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:10.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.265+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 shutdown_connections 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.265+0000 7fe44beaf640 1 -- 172.21.15.22:0/375674690 wait complete. 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 10 monclient: build_initial_monmap 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 10 monclient: monmap: 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:10.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 1 librados: starting msgr at 2024-03-29T11:01:10.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 1 librados: starting objecter 2024-03-29T11:01:10.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 1 Processor -- start 2024-03-29T11:01:10.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 1 -- start start 2024-03-29T11:01:10.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 1 librados: setting wanted keys 2024-03-29T11:01:10.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 1 librados: calling monclient init 2024-03-29T11:01:10.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.266+0000 7fe44beaf640 10 monclient: init 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.267+0000 7fe44beaf640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.267+0000 7fe44beaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.267+0000 7fe44beaf640 10 monclient(hunting): picked mon.noname-a con 0x7fe4440b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44beaf640 10 monclient(hunting): picked mon.noname-b con 0x7fe4441422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44beaf640 10 monclient(hunting): picked mon.noname-c con 0x7fe444142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44a44c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fe444142690 0x7fe4441e1a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57686/0 (socket says 172.21.15.22:57686) 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44a44c640 1 -- 172.21.15.22:0/762571540 learned_addr learned my addr 172.21.15.22:0/762571540 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440adc90 con 0x7fe4440b17d0 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440adf70 con 0x7fe4441422b0 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4440aba40 con 0x7fe444142690 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44beaf640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.268+0000 7fe44beaf640 10 monclient(hunting): authenticate will time out at 1171.574060s 2024-03-29T11:01:10.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 client.?.objecter ms_handle_connect 0x7fe444142690 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 client.?.objecter ms_handle_connect 0x7fe4441422b0 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4193964491 0 0) 0x7fe4440aba40 con 0x7fe444142690 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428004eb0 con 0x7fe444142690 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1059586557 0 0) 0x7fe428004eb0 con 0x7fe444142690 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4440aba40 con 0x7fe444142690 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe43c001f20 con 0x7fe444142690 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:10.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): dump: 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155831723 0 0) 0x7fe4440adf70 con 0x7fe4441422b0 2024-03-29T11:01:10.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 10 monclient(hunting): my global_id is 4432 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe428004eb0 con 0x7fe4441422b0 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 650957548 0 0) 0x7fe4440aba40 con 0x7fe444142690 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 >> v1:172.21.15.26:6789/0 conn(0x7fe4441422b0 legacy=0x7fe4440ae420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.269+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 >> v1:172.21.15.22:6789/0 conn(0x7fe4440b17d0 legacy=0x7fe4440ac6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 1 monclient: found mon.c 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4440b6740 con 0x7fe444142690 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:01:10.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe44beaf640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe44beaf640 10 monclient: _renew_subs 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe44beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4440aeb30 con 0x7fe444142690 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe43c004e10 con 0x7fe444142690 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fe43c005220 con 0x7fe444142690 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43a7fc640 10 monclient: dump: 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.258 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.270+0000 7fe43bfff640 4 set_mon_vals no callback set 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.271+0000 7fe44beaf640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.271+0000 7fe44beaf640 10 monclient: _renew_subs 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.271+0000 7fe44beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.271+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4440ac140 con 0x7fe444142690 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.271+0000 7fe44beaf640 1 librados: init done 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.272+0000 7fe44beaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:10.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.273+0000 7fe44beaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:10.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.273+0000 7fe44beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:10.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.273+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4440ab8a0 con 0x7fe444142690 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.273+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fe43c005380 con 0x7fe444142690 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.274+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fe43c085050 con 0x7fe444142690 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.274+0000 7fe43a7fc640 10 client.4559.objecter ms_dispatch 0x7fe444000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.274+0000 7fe43a7fc640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.274+0000 7fe43a7fc640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.274+0000 7fe43a7fc640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.274+0000 7fe449c4b640 10 monclient: get_auth_request con 0x7fe428076540 auth_method 0 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.275+0000 7fe43a7fc640 10 client.4559.objecter ms_handle_connect 0x7fe428076540 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.280+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4440ab8a0 con 0x7fe444142690 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.280+0000 7fe43a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:10.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.280+0000 7fe43a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:10.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.599+0000 7fe44beaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:10.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.599+0000 7fe44beaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:10.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.599+0000 7fe44beaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:10.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.599+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4440ab8a0 con 0x7fe444142690 2024-03-29T11:01:10.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.601+0000 7fe43a7fc640 1 -- 172.21.15.22:0/762571540 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fe4440ab8a0 con 0x7fe444142690 2024-03-29T11:01:10.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.601+0000 7fe43a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:10.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.601+0000 7fe43a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:10.582 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:10.582 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:10.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 10 librados: watch_flush enter 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 10 librados: watch_flush exit 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 >> v1:172.21.15.22:6816/380221644 conn(0x7fe428076540 legacy=0x7fe428078a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 10 monclient: shutdown 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 >> v1:172.21.15.22:6790/0 conn(0x7fe444142690 legacy=0x7fe4441e1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 shutdown_connections 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.603+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 >> 172.21.15.22:0/762571540 conn(0x7fe444066a00 msgr2=0x7fe4440b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.604+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 shutdown_connections 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.604+0000 7fe44beaf640 1 -- 172.21.15.22:0/762571540 wait complete. 2024-03-29T11:01:10.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:10.604+0000 7fe44beaf640 1 librados: shutdown 2024-03-29T11:01:10.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:17.405 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.578+0000 7fafe5915640 10 monclient: get_monmap_and_config 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.578+0000 7fafe5915640 10 monclient: build_initial_monmap 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.578+0000 7fafe5915640 10 monclient: monmap: 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:17.580130+0000 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:17.580130+0000 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:17.559 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:17.560 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:17.560 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:17.560 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:17.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.582+0000 7fafe5915640 1 Processor -- start 2024-03-29T11:01:17.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.582+0000 7fafe5915640 1 -- start start 2024-03-29T11:01:17.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.582+0000 7fafe5915640 10 monclient: init 2024-03-29T11:01:17.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:17.563 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 10 monclient(hunting): picked mon.noname-a con 0x7fafe01422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 10 monclient(hunting): picked mon.noname-b con 0x7fafe0142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 10 monclient(hunting): picked mon.noname-c con 0x7fafe00b17d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0142c80 con 0x7fafe01422b0 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe00aae40 con 0x7fafe00b17d0 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe013fbd0 con 0x7fafe0142690 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafe5915640 10 monclient(hunting): authenticate will time out at 1178.888244s 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafde7fc640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fafe0142690 0x7fafe00b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52116/0 (socket says 172.21.15.22:52116) 2024-03-29T11:01:17.564 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.583+0000 7fafde7fc640 1 -- 172.21.15.22:0/3670118224 learned_addr learned my addr 172.21.15.22:0/3670118224 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.586+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440246204 0 0) 0x7fafe013fbd0 con 0x7fafe0142690 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.586+0000 7fafddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc8003520 con 0x7fafe0142690 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4060678756 0 0) 0x7fafe0142c80 con 0x7fafe01422b0 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:17.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafe013fbd0 con 0x7fafe01422b0 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3580582273 0 0) 0x7fafc8003520 con 0x7fafe0142690 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 >> v1:172.21.15.26:6789/0 conn(0x7fafe00b17d0 legacy=0x7fafe00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 >> v1:172.21.15.22:6789/0 conn(0x7fafe01422b0 legacy=0x7fafe013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 monclient: found mon.noname-b 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe0151940 con 0x7fafe0142690 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:17.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.587+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fafd8001a10 con 0x7fafe0142690 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.588+0000 7fafddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.588+0000 7fafddffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.588+0000 7fafddffb640 10 monclient: dump: 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:17.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.588+0000 7fafe5915640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:01:17.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.588+0000 7fafe5915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:17.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.589+0000 7fafddffb640 1 -- 172.21.15.22:0/3670118224 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd8001b70 con 0x7fafe0142690 2024-03-29T11:01:17.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.589+0000 7fafddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:17.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.589+0000 7fafe5915640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:17.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.589+0000 7fafe5915640 4 set_mon_vals no callback set 2024-03-29T11:01:17.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.590+0000 7fafe5915640 10 monclient: shutdown 2024-03-29T11:01:17.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.590+0000 7fafe5915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.590+0000 7fafe5915640 1 -- 172.21.15.22:0/3670118224 >> v1:172.21.15.22:6790/0 conn(0x7fafe0142690 legacy=0x7fafe00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.591+0000 7fafe5915640 1 -- 172.21.15.22:0/3670118224 shutdown_connections 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.591+0000 7fafe5915640 1 -- 172.21.15.22:0/3670118224 >> 172.21.15.22:0/3670118224 conn(0x7fafe0066a00 msgr2=0x7fafe0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.591+0000 7fafe5915640 1 -- 172.21.15.22:0/3670118224 shutdown_connections 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.592+0000 7fafe5915640 1 -- 172.21.15.22:0/3670118224 wait complete. 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.592+0000 7fafe5915640 10 monclient: build_initial_monmap 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.592+0000 7fafe5915640 10 monclient: monmap: 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:17.574 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:17.575 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:17.575 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:17.575 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:17.575 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:17.575 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:17.575 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:17.575 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:17.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.593+0000 7fafe5915640 1 librados: starting msgr at 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.593+0000 7fafe5915640 1 librados: starting objecter 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.593+0000 7fafe5915640 1 Processor -- start 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.593+0000 7fafe5915640 1 -- start start 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.593+0000 7fafe5915640 1 librados: setting wanted keys 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.593+0000 7fafe5915640 1 librados: calling monclient init 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.593+0000 7fafe5915640 10 monclient: init 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:17.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 10 monclient(hunting): picked mon.noname-c con 0x7fafe00b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 10 monclient(hunting): picked mon.noname-b con 0x7fafe01422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 10 monclient(hunting): picked mon.noname-a con 0x7fafe0142690 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe006e9b0 con 0x7fafe0142690 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0070a50 con 0x7fafe01422b0 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe0070d30 con 0x7fafe00b17d0 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafe5915640 10 monclient(hunting): authenticate will time out at 1178.899711s 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafdeffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fafe00b17d0 0x7fafe006ee40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53424/0 (socket says 172.21.15.22:53424) 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.594+0000 7fafdeffd640 1 -- 172.21.15.22:0/2398657294 learned_addr learned my addr 172.21.15.22:0/2398657294 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 client.?.objecter ms_handle_connect 0x7fafe00b17d0 2024-03-29T11:01:17.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:17.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975115213 0 0) 0x7fafe0070d30 con 0x7fafe00b17d0 2024-03-29T11:01:17.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:17.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:17.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:17.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:01:17.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafcc004eb0 con 0x7fafe00b17d0 2024-03-29T11:01:17.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 619207361 0 0) 0x7fafcc004eb0 con 0x7fafe00b17d0 2024-03-29T11:01:17.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fafe0070d30 con 0x7fafe00b17d0 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fafd4001a10 con 0x7fafe00b17d0 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): dump: 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:17.585 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3269901231 0 0) 0x7fafe0070d30 con 0x7fafe00b17d0 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.595+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 >> v1:172.21.15.26:6789/0 conn(0x7fafe01422b0 legacy=0x7fafe006f550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.598+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 >> v1:172.21.15.22:6789/0 conn(0x7fafe0142690 legacy=0x7fafe01e6270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.598+0000 7fafc7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafc7fff640 1 monclient: found mon.c 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafc7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafc7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe00717a0 con 0x7fafe00b17d0 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafc7fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:17.586 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafc7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 10 monclient: _renew_subs 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 1 -- 172.21.15.22:0/2398657294 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe00719f0 con 0x7fafe00b17d0 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 10 monclient: _renew_subs 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 1 -- 172.21.15.22:0/2398657294 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe00b0a70 con 0x7fafe00b17d0 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.599+0000 7fafe5915640 1 librados: init done 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.600+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fafd4001b70 con 0x7fafe00b17d0 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.600+0000 7fafc7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:17.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.600+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fafd4003a60 con 0x7fafe00b17d0 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.600+0000 7fafc7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.600+0000 7fafc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.600+0000 7fafc7fff640 10 monclient: dump: 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:17.588 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.601+0000 7fafc5ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.601+0000 7fafc5ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.601+0000 7fafc5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.601+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc0005080 con 0x7fafe00b17d0 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.602+0000 7fafdd7fa640 4 set_mon_vals no callback set 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.602+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fafd4003bc0 con 0x7fafe00b17d0 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.603+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fafd4084c10 con 0x7fafe00b17d0 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.603+0000 7fafc7fff640 10 client.4565.objecter ms_dispatch 0x7fafe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.603+0000 7fafc7fff640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.603+0000 7fafc7fff640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.603+0000 7fafc7fff640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.603+0000 7fafde7fc640 10 monclient: get_auth_request con 0x7fafcc06dc80 auth_method 0 2024-03-29T11:01:17.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.603+0000 7fafc7fff640 10 client.4565.objecter ms_handle_connect 0x7fafcc06dc80 2024-03-29T11:01:17.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.611+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fafd4050320 con 0x7fafe00b17d0 2024-03-29T11:01:17.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.611+0000 7fafc7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:17.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.611+0000 7fafc7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:17.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.955+0000 7fafc5ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:17.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.955+0000 7fafc5ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:17.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.955+0000 7fafc5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:17.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.955+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafc0005330 con 0x7fafe00b17d0 2024-03-29T11:01:17.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.956+0000 7fafc7fff640 1 -- 172.21.15.22:0/2398657294 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fafd4052420 con 0x7fafe00b17d0 2024-03-29T11:01:17.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.957+0000 7fafc7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:17.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.957+0000 7fafc7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:17.937 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:17.938 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:17.940 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 10 librados: watch_flush enter 2024-03-29T11:01:17.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 10 librados: watch_flush exit 2024-03-29T11:01:17.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:01:17.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 >> v1:172.21.15.22:6816/380221644 conn(0x7fafcc06dc80 legacy=0x7fafcc070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 10 monclient: shutdown 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 >> v1:172.21.15.22:6790/0 conn(0x7fafe00b17d0 legacy=0x7fafe006ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 shutdown_connections 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.960+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 >> 172.21.15.22:0/2398657294 conn(0x7fafe0066a00 msgr2=0x7fafe013e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.961+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 shutdown_connections 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.961+0000 7fafc5ffb640 1 -- 172.21.15.22:0/2398657294 wait complete. 2024-03-29T11:01:17.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:17.961+0000 7fafc5ffb640 1 librados: shutdown 2024-03-29T11:01:17.961 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:24.767 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:24.913 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.933+0000 7ffabffff640 10 monclient: get_monmap_and_config 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.933+0000 7ffabffff640 10 monclient: build_initial_monmap 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.933+0000 7ffabffff640 10 monclient: monmap: 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:24.934647+0000 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:24.934647+0000 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:24.914 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:24.916 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.937+0000 7ffabffff640 1 Processor -- start 2024-03-29T11:01:24.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.937+0000 7ffabffff640 1 -- start start 2024-03-29T11:01:24.917 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.937+0000 7ffabffff640 10 monclient: init 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 10 monclient(hunting): picked mon.noname-c con 0x7ffac00a9750 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 10 monclient(hunting): picked mon.noname-a con 0x7ffac00add40 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 10 monclient(hunting): picked mon.noname-b con 0x7ffac00aad20 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0149870 con 0x7ffac00add40 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac00abf80 con 0x7ffac00a9750 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0149ac0 con 0x7ffac00aad20 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.938+0000 7ffabffff640 10 monclient(hunting): authenticate will time out at 1186.242525s 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.939+0000 7ffabf7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ffac00aad20 0x7ffac0151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:36808/0 (socket says 172.21.15.22:36808) 2024-03-29T11:01:24.919 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.939+0000 7ffabf7fe640 1 -- 172.21.15.22:0/1587411905 learned_addr learned my addr 172.21.15.22:0/1587411905 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:24.920 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055964050 0 0) 0x7ffac0149ac0 con 0x7ffac00aad20 2024-03-29T11:01:24.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:24.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:24.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:24.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:01:24.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaa8003520 con 0x7ffac00aad20 2024-03-29T11:01:24.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3556171227 0 0) 0x7ffaa8003520 con 0x7ffac00aad20 2024-03-29T11:01:24.921 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.941+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 >> v1:172.21.15.26:6789/0 conn(0x7ffac00a9750 legacy=0x7ffac0149fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 >> v1:172.21.15.22:6789/0 conn(0x7ffac00add40 legacy=0x7ffac0143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 1 monclient: found mon.noname-b 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac014a6f0 con 0x7ffac00aad20 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:24.924 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffab4001690 con 0x7ffac00aad20 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.943+0000 7ffabdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabdffb640 10 monclient: dump: 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabffff640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:01:24.925 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabffff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:24.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabdffb640 1 -- 172.21.15.22:0/1587411905 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab4001920 con 0x7ffac00aad20 2024-03-29T11:01:24.926 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabffff640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.944+0000 7ffabffff640 4 set_mon_vals no callback set 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.945+0000 7ffabffff640 10 monclient: shutdown 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.945+0000 7ffabffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.945+0000 7ffabffff640 1 -- 172.21.15.22:0/1587411905 >> v1:172.21.15.22:6790/0 conn(0x7ffac00aad20 legacy=0x7ffac0151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.946+0000 7ffabffff640 1 -- 172.21.15.22:0/1587411905 shutdown_connections 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.946+0000 7ffabffff640 1 -- 172.21.15.22:0/1587411905 >> 172.21.15.22:0/1587411905 conn(0x7ffac00673e0 msgr2=0x7ffac00b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:24.927 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.947+0000 7ffabffff640 1 -- 172.21.15.22:0/1587411905 shutdown_connections 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.949+0000 7ffabffff640 1 -- 172.21.15.22:0/1587411905 wait complete. 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.950+0000 7ffabffff640 10 monclient: build_initial_monmap 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.950+0000 7ffabffff640 10 monclient: monmap: 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:24.930 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.950+0000 7ffabffff640 1 librados: starting msgr at 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.951+0000 7ffabffff640 1 librados: starting objecter 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.951+0000 7ffabffff640 1 Processor -- start 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.951+0000 7ffabffff640 1 -- start start 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.951+0000 7ffabffff640 1 librados: setting wanted keys 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.951+0000 7ffabffff640 1 librados: calling monclient init 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.951+0000 7ffabffff640 10 monclient: init 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 10 monclient(hunting): picked mon.noname-c con 0x7ffac00a9750 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 10 monclient(hunting): picked mon.noname-a con 0x7ffac00aad20 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:24.934 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 10 monclient(hunting): picked mon.noname-b con 0x7ffac00add40 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:24.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0147b10 con 0x7ffac00aad20 2024-03-29T11:01:24.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac01474c0 con 0x7ffac00add40 2024-03-29T11:01:24.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac0144a00 con 0x7ffac00a9750 2024-03-29T11:01:24.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:24.935 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.952+0000 7ffabffff640 10 monclient(hunting): authenticate will time out at 1186.257019s 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.954+0000 7ffabeffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ffac00a9750 0x7ffac0071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37304/0 (socket says 172.21.15.22:37304) 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffabeffd640 1 -- 172.21.15.22:0/859260928 learned_addr learned my addr 172.21.15.22:0/859260928 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 client.?.objecter ms_handle_connect 0x7ffac00a9750 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 client.?.objecter ms_handle_connect 0x7ffac00add40 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087187127 0 0) 0x7ffac01474c0 con 0x7ffac00add40 2024-03-29T11:01:24.936 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 10 monclient(hunting): my global_id is 4435 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.955+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaac004eb0 con 0x7ffac00add40 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545760128 0 0) 0x7ffac0144a00 con 0x7ffac00a9750 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaac0051d0 con 0x7ffac00a9750 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3815084047 0 0) 0x7ffaac0051d0 con 0x7ffac00a9750 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffac0144a00 con 0x7ffac00a9750 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffab8001bf0 con 0x7ffac00a9750 2024-03-29T11:01:24.937 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): dump: 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1834062682 0 0) 0x7ffac0144a00 con 0x7ffac00a9750 2024-03-29T11:01:24.938 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.956+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 >> v1:172.21.15.26:6789/0 conn(0x7ffac00add40 legacy=0x7ffac01486b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:24.939 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.958+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 >> v1:172.21.15.22:6789/0 conn(0x7ffac00aad20 legacy=0x7ffac0147fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 1 monclient: found mon.c 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac01476c0 con 0x7ffac00a9750 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 10 client.?.objecter ms_handle_connect 0x7ffac00aad20 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffaa7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:01:24.941 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 10 monclient: _renew_subs 2024-03-29T11:01:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac01df770 con 0x7ffac00a9750 2024-03-29T11:01:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 10 monclient: _renew_subs 2024-03-29T11:01:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac01def10 con 0x7ffac00a9750 2024-03-29T11:01:24.942 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.961+0000 7ffabffff640 1 librados: init done 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.962+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffab8002100 con 0x7ffac00a9750 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.962+0000 7ffaa7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.962+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ffab8004e20 con 0x7ffac00a9750 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.962+0000 7ffaa7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.962+0000 7ffaa7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.962+0000 7ffaa7fff640 10 monclient: dump: 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:24.943 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.962+0000 7ffabd7fa640 4 set_mon_vals no callback set 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.963+0000 7ffabffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.963+0000 7ffabffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.963+0000 7ffabffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:24.944 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.963+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaa0005080 con 0x7ffac00a9750 2024-03-29T11:01:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.964+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7ffab801adf0 con 0x7ffac00a9750 2024-03-29T11:01:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.965+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7ffab8084de0 con 0x7ffac00a9750 2024-03-29T11:01:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.965+0000 7ffaa7fff640 10 client.4571.objecter ms_dispatch 0x7ffac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.965+0000 7ffaa7fff640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.965+0000 7ffaa7fff640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.965+0000 7ffaa7fff640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:01:24.945 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.966+0000 7ffabe7fc640 10 monclient: get_auth_request con 0x7ffaac06de40 auth_method 0 2024-03-29T11:01:24.946 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.967+0000 7ffaa7fff640 10 client.4571.objecter ms_handle_connect 0x7ffaac06de40 2024-03-29T11:01:24.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.972+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffab80506c0 con 0x7ffac00a9750 2024-03-29T11:01:24.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.972+0000 7ffaa7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:24.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:24.972+0000 7ffaa7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:25.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.297+0000 7ffabffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:25.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.297+0000 7ffabffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:25.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.297+0000 7ffabffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:25.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.297+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaa0005330 con 0x7ffac00a9750 2024-03-29T11:01:25.277 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.298+0000 7ffaa7fff640 1 -- 172.21.15.22:0/859260928 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7ffab80525f0 con 0x7ffac00a9750 2024-03-29T11:01:25.278 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:25.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.298+0000 7ffaa7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:25.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.298+0000 7ffaa7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:25.278 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:25.281 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.301+0000 7ffabffff640 10 librados: watch_flush enter 2024-03-29T11:01:25.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.301+0000 7ffabffff640 10 librados: watch_flush exit 2024-03-29T11:01:25.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:01:25.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 >> v1:172.21.15.22:6816/380221644 conn(0x7ffaac06de40 legacy=0x7ffaac070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 10 monclient: shutdown 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 >> v1:172.21.15.22:6790/0 conn(0x7ffac00a9750 legacy=0x7ffac0071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 shutdown_connections 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 >> 172.21.15.22:0/859260928 conn(0x7ffac00673e0 msgr2=0x7ffac01418a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.302+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 shutdown_connections 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.303+0000 7ffabffff640 1 -- 172.21.15.22:0/859260928 wait complete. 2024-03-29T11:01:25.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:25.303+0000 7ffabffff640 1 librados: shutdown 2024-03-29T11:01:25.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:32.106 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:32.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.281+0000 7f0cd4ef6640 10 monclient: get_monmap_and_config 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.281+0000 7f0cd4ef6640 10 monclient: build_initial_monmap 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.281+0000 7f0cd4ef6640 10 monclient: monmap: 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:32.282738+0000 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:32.282738+0000 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:32.262 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:32.263 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:32.263 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.282+0000 7f0cd4ef6640 1 Processor -- start 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.283+0000 7f0cd4ef6640 1 -- start start 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.283+0000 7f0cd4ef6640 10 monclient: init 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f0cd00b1f60 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f0cd0071eb0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cce59c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0cd00b1f60 0x7f0cd0143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59448/0 (socket says 172.21.15.22:59448) 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f0cd0143d10 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd006ef80 con 0x7f0cd0143d10 2024-03-29T11:01:32.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd006f260 con 0x7f0cd0071eb0 2024-03-29T11:01:32.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd00b40c0 con 0x7f0cd00b1f60 2024-03-29T11:01:32.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:32.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cd4ef6640 10 monclient(hunting): authenticate will time out at 1193.588894s 2024-03-29T11:01:32.266 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.284+0000 7f0cce59c640 1 -- 172.21.15.22:0/3917303855 learned_addr learned my addr 172.21.15.22:0/3917303855 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:32.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.286+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202230539 0 0) 0x7f0cd00b40c0 con 0x7f0cd00b1f60 2024-03-29T11:01:32.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.286+0000 7f0ccd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:01:32.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.286+0000 7f0ccd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:32.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.286+0000 7f0ccd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:32.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.286+0000 7f0ccd59a640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:01:32.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.286+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb8003520 con 0x7f0cd00b1f60 2024-03-29T11:01:32.267 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.287+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729007007 0 0) 0x7f0cd006ef80 con 0x7f0cd0143d10 2024-03-29T11:01:32.268 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.287+0000 7f0ccd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.287+0000 7f0ccd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.287+0000 7f0ccd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.287+0000 7f0ccd59a640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.287+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd00b40c0 con 0x7f0cd0143d10 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2182582443 0 0) 0x7f0cd00b40c0 con 0x7f0cd0143d10 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 >> v1:172.21.15.22:6790/0 conn(0x7f0cd00b1f60 legacy=0x7f0cd0143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 >> v1:172.21.15.26:6789/0 conn(0x7f0cd0071eb0 legacy=0x7f0cd0146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:01:32.272 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 monclient: found mon.noname-a 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0155d70 con 0x7f0cd0143d10 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: _finish_auth 0 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0cc80019f0 con 0x7f0cd0143d10 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: dump: 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.273 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0cd4ef6640 5 monclient: authenticate success, global_id 4449 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0cd4ef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cc8001b50 con 0x7f0cd0143d10 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 1 -- 172.21.15.22:0/3917303855 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0cc8001fa0 con 0x7f0cd0143d10 2024-03-29T11:01:32.274 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0ccd59a640 10 monclient: dump: 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0cd4ef6640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:32.275 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.288+0000 7f0cd4ef6640 4 set_mon_vals no callback set 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.289+0000 7f0cd4ef6640 10 monclient: shutdown 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.289+0000 7f0cd4ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.289+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/3917303855 >> v1:172.21.15.22:6789/0 conn(0x7f0cd0143d10 legacy=0x7f0cd0146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.289+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/3917303855 shutdown_connections 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.289+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/3917303855 >> 172.21.15.22:0/3917303855 conn(0x7f0cd0066a00 msgr2=0x7f0cd0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.291+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/3917303855 shutdown_connections 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.291+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/3917303855 wait complete. 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.292+0000 7f0cd4ef6640 10 monclient: build_initial_monmap 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.292+0000 7f0cd4ef6640 10 monclient: monmap: 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:32.276 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:32.277 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:32.277 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:32.277 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:32.277 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.293+0000 7f0cd4ef6640 1 librados: starting msgr at 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.293+0000 7f0cd4ef6640 1 librados: starting objecter 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.293+0000 7f0cd4ef6640 1 Processor -- start 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.295+0000 7f0cd4ef6640 1 -- start start 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.295+0000 7f0cd4ef6640 1 librados: setting wanted keys 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.295+0000 7f0cd4ef6640 1 librados: calling monclient init 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.295+0000 7f0cd4ef6640 10 monclient: init 2024-03-29T11:01:32.278 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.296+0000 7f0cd4ef6640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:32.279 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.296+0000 7f0cd4ef6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:01:32.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.296+0000 7f0cd4ef6640 10 monclient(hunting): picked mon.noname-a con 0x7f0cd00b1f60 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:32.282 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.299+0000 7f0cd4ef6640 10 monclient(hunting): picked mon.noname-b con 0x7f0cd0143d10 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.299+0000 7f0cd4ef6640 10 monclient(hunting): picked mon.noname-c con 0x7f0cd0142180 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.299+0000 7f0cced9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0cd0142180 0x7f0cd01e63a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60978/0 (socket says 172.21.15.22:60978) 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.299+0000 7f0cced9d640 1 -- 172.21.15.22:0/2633469422 learned_addr learned my addr 172.21.15.22:0/2633469422 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd006f640 con 0x7f0cd00b1f60 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd01347c0 con 0x7f0cd0143d10 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd01547d0 con 0x7f0cd0142180 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cd4ef6640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cd4ef6640 10 monclient(hunting): authenticate will time out at 1193.604160s 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 client.?.objecter ms_handle_connect 0x7f0cd0142180 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:32.283 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 client.?.objecter ms_handle_connect 0x7f0cd00b1f60 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280692952 0 0) 0x7f0cd01547d0 con 0x7f0cd0142180 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cb0004eb0 con 0x7f0cd0142180 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1415966886 0 0) 0x7f0cd006f640 con 0x7f0cd00b1f60 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:01:32.284 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.300+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd01547d0 con 0x7f0cd00b1f60 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3231444899 0 0) 0x7f0cb0004eb0 con 0x7f0cd0142180 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cd006f640 con 0x7f0cd0142180 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1073951393 0 0) 0x7f0cd01547d0 con 0x7f0cd00b1f60 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cb0004eb0 con 0x7f0cd00b1f60 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0cc80019b0 con 0x7f0cd0142180 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 10 monclient(hunting): dump: 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.285 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0cc00018b0 con 0x7f0cd00b1f60 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 10 monclient(hunting): dump: 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.286 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3241420830 0 0) 0x7f0cd006f640 con 0x7f0cd0142180 2024-03-29T11:01:32.287 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.301+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 >> v1:172.21.15.26:6789/0 conn(0x7f0cd0143d10 legacy=0x7f0cd01e2c70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.304+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 >> v1:172.21.15.22:6789/0 conn(0x7f0cd00b1f60 legacy=0x7f0cd01330a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.304+0000 7f0cbeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.304+0000 7f0cbeffd640 1 monclient: found mon.c 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.304+0000 7f0cbeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.304+0000 7f0cbeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.304+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd0154a40 con 0x7f0cd0142180 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cbeffd640 10 monclient: _finish_auth 0 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cbeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cbeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:01:32.290 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 10 monclient: _renew_subs 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd0153500 con 0x7f0cd0142180 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 10 monclient: _renew_subs 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd0142f00 con 0x7f0cd0142180 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.305+0000 7f0cd4ef6640 1 librados: init done 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.307+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cc8003c20 con 0x7f0cd0142180 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.307+0000 7f0cbeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.307+0000 7f0cccd99640 4 set_mon_vals no callback set 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.308+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0cc8003f70 con 0x7f0cd0142180 2024-03-29T11:01:32.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.308+0000 7f0cbeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.308+0000 7f0cbeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.308+0000 7f0cbeffd640 10 monclient: dump: 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:32.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.308+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f0cc8004190 con 0x7f0cd0142180 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cd4ef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cd4ef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cd4ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cb4005080 con 0x7f0cd0142180 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0cc8084e20 con 0x7f0cd0142180 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cbeffd640 10 client.4577.objecter ms_dispatch 0x7f0cd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cbeffd640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cbeffd640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:32.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.309+0000 7f0cbeffd640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:01:32.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.311+0000 7f0cce59c640 10 monclient: get_auth_request con 0x7f0cb006df90 auth_method 0 2024-03-29T11:01:32.294 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.311+0000 7f0cbeffd640 10 client.4577.objecter ms_handle_connect 0x7f0cb006df90 2024-03-29T11:01:32.296 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.317+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0cc80507c0 con 0x7f0cd0142180 2024-03-29T11:01:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.317+0000 7f0cbeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:32.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.317+0000 7f0cbeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:32.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.653+0000 7f0cd4ef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:32.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.653+0000 7f0cd4ef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:32.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.653+0000 7f0cd4ef6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:32.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.653+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cb4005330 con 0x7f0cd0142180 2024-03-29T11:01:32.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.655+0000 7f0cbeffd640 1 -- 172.21.15.22:0/2633469422 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0cc80526f0 con 0x7f0cd0142180 2024-03-29T11:01:32.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.655+0000 7f0cbeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:32.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.655+0000 7f0cbeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:32.636 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:32.636 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:32.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.659+0000 7f0cd4ef6640 10 librados: watch_flush enter 2024-03-29T11:01:32.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.659+0000 7f0cd4ef6640 10 librados: watch_flush exit 2024-03-29T11:01:32.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:01:32.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:01:32.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 >> v1:172.21.15.22:6816/380221644 conn(0x7f0cb006df90 legacy=0x7f0cb0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:32.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 10 monclient: shutdown 2024-03-29T11:01:32.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:32.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 >> v1:172.21.15.22:6790/0 conn(0x7f0cd0142180 legacy=0x7f0cd01e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:32.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 shutdown_connections 2024-03-29T11:01:32.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 >> 172.21.15.22:0/2633469422 conn(0x7f0cd0066a00 msgr2=0x7f0cd00b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:32.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 shutdown_connections 2024-03-29T11:01:32.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.660+0000 7f0cd4ef6640 1 -- 172.21.15.22:0/2633469422 wait complete. 2024-03-29T11:01:32.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:32.663+0000 7f0cd4ef6640 1 librados: shutdown 2024-03-29T11:01:32.661 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:39.468 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:39.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.656+0000 7f01fbfff640 10 monclient: get_monmap_and_config 2024-03-29T11:01:39.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.656+0000 7f01fbfff640 10 monclient: build_initial_monmap 2024-03-29T11:01:39.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.656+0000 7f01fbfff640 10 monclient: monmap: 2024-03-29T11:01:39.637 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:39.637 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:39.637 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:39.657337+0000 2024-03-29T11:01:39.638 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:39.657337+0000 2024-03-29T11:01:39.638 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:39.638 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:39.638 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:39.638 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:39.638 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:39.638 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:39.639 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.658+0000 7f01fbfff640 1 Processor -- start 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.658+0000 7f01fbfff640 1 -- start start 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.658+0000 7f01fbfff640 10 monclient: init 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f01fc0a9750 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f01fc0af100 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f01fc13d200 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01fc06dc20 con 0x7f01fc0a9750 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01fc06df00 con 0x7f01fc13d200 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01fc13d710 con 0x7f01fc0af100 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:39.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.659+0000 7f01fbfff640 10 monclient(hunting): authenticate will time out at 1200.963628s 2024-03-29T11:01:39.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.662+0000 7f01fb7fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f01fc13d200 0x7f01fc143960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:33779/0 (socket says 172.21.15.22:33779) 2024-03-29T11:01:39.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.662+0000 7f01fb7fe640 1 -- 172.21.15.22:0/2044820136 learned_addr learned my addr 172.21.15.22:0/2044820136 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907893291 0 0) 0x7f01fc13d710 con 0x7f01fc0af100 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e4003520 con 0x7f01fc0af100 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312038090 0 0) 0x7f01fc06dc20 con 0x7f01fc0a9750 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:39.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 10 monclient(hunting): my global_id is 4455 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.663+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01fc13d710 con 0x7f01fc0a9750 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1923610882 0 0) 0x7f01fc06df00 con 0x7f01fc13d200 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01fc06dc20 con 0x7f01fc13d200 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797755012 0 0) 0x7f01e4003520 con 0x7f01fc0af100 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 >> v1:172.21.15.26:6789/0 conn(0x7f01fc13d200 legacy=0x7f01fc143960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 >> v1:172.21.15.22:6789/0 conn(0x7f01fc0a9750 legacy=0x7f01fc13caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 monclient: found mon.noname-b 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:39.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:01:39.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc0af650 con 0x7f01fc0af100 2024-03-29T11:01:39.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:39.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:39.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:39.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01fbfff640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:01:39.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01fbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f01f4001940 con 0x7f01fc0af100 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: dump: 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:39.647 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01fbfff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01f4001b60 con 0x7f01fc0af100 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 1 -- 172.21.15.22:0/2044820136 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f01f4001ef0 con 0x7f01fc0af100 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01f9ffb640 10 monclient: dump: 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.648 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01fbfff640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.664+0000 7f01fbfff640 4 set_mon_vals no callback set 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.665+0000 7f01fbfff640 10 monclient: shutdown 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.665+0000 7f01fbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.665+0000 7f01fbfff640 1 -- 172.21.15.22:0/2044820136 >> v1:172.21.15.22:6790/0 conn(0x7f01fc0af100 legacy=0x7f01fc143250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.666+0000 7f01fbfff640 1 -- 172.21.15.22:0/2044820136 shutdown_connections 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.666+0000 7f01fbfff640 1 -- 172.21.15.22:0/2044820136 >> 172.21.15.22:0/2044820136 conn(0x7f01fc0b3a30 msgr2=0x7f01fc0b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.667+0000 7f01fbfff640 1 -- 172.21.15.22:0/2044820136 shutdown_connections 2024-03-29T11:01:39.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.667+0000 7f01fbfff640 1 -- 172.21.15.22:0/2044820136 wait complete. 2024-03-29T11:01:39.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.668+0000 7f01fbfff640 10 monclient: build_initial_monmap 2024-03-29T11:01:39.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.668+0000 7f01fbfff640 10 monclient: monmap: 2024-03-29T11:01:39.651 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.669+0000 7f01fbfff640 1 librados: starting msgr at 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.669+0000 7f01fbfff640 1 librados: starting objecter 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.669+0000 7f01fbfff640 1 Processor -- start 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.670+0000 7f01fbfff640 1 -- start start 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.670+0000 7f01fbfff640 1 librados: setting wanted keys 2024-03-29T11:01:39.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.670+0000 7f01fbfff640 1 librados: calling monclient init 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.670+0000 7f01fbfff640 10 monclient: init 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f01fc0a9750 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 10 monclient(hunting): picked mon.noname-a con 0x7f01fc0af100 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f01fc13d200 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01fc072360 con 0x7f01fc0af100 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01fc071530 con 0x7f01fc13d200 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01fc1dbd10 con 0x7f01fc0a9750 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01faffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f01fc0a9750 0x7f01fc071c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57986/0 (socket says 172.21.15.22:57986) 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01faffd640 1 -- 172.21.15.22:0/246310263 learned_addr learned my addr 172.21.15.22:0/246310263 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:39.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.671+0000 7f01fbfff640 10 monclient(hunting): authenticate will time out at 1200.975802s 2024-03-29T11:01:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.674+0000 7f01e37fe640 10 client.?.objecter ms_handle_connect 0x7f01fc0a9750 2024-03-29T11:01:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.674+0000 7f01e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:39.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.674+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1418212715 0 0) 0x7f01fc1dbd10 con 0x7f01fc0a9750 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01e8004eb0 con 0x7f01fc0a9750 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 10 client.?.objecter ms_handle_connect 0x7f01fc0af100 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2278995501 0 0) 0x7f01e8004eb0 con 0x7f01fc0a9750 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01fc1dbd10 con 0x7f01fc0a9750 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f01f0001a30 con 0x7f01fc0a9750 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.675+0000 7f01e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.676+0000 7f01e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.676+0000 7f01e37fe640 10 monclient(hunting): dump: 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:39.656 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:39.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.676+0000 7f01e37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 10 client.?.objecter ms_handle_connect 0x7f01fc13d200 2024-03-29T11:01:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:39.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 309958935 0 0) 0x7f01fc1dbd10 con 0x7f01fc0a9750 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 >> v1:172.21.15.26:6789/0 conn(0x7f01fc13d200 legacy=0x7f01fc1e5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 >> v1:172.21.15.22:6789/0 conn(0x7f01fc0af100 legacy=0x7f01fc06ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 1 monclient: found mon.c 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.678+0000 7f01e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01e37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01fc1dc230 con 0x7f01fc0a9750 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01e37fe640 10 monclient: _finish_auth 0 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01e37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 10 monclient: _renew_subs 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:39.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 1 -- 172.21.15.22:0/246310263 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01fc1dc480 con 0x7f01fc0a9750 2024-03-29T11:01:39.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:39.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 10 monclient: _renew_subs 2024-03-29T11:01:39.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:39.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 1 -- 172.21.15.22:0/246310263 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01fc1db790 con 0x7f01fc0a9750 2024-03-29T11:01:39.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.679+0000 7f01fbfff640 1 librados: init done 2024-03-29T11:01:39.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.682+0000 7f01e17fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:39.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.682+0000 7f01e17fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:39.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.682+0000 7f01e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:39.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.682+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01fc04f730 con 0x7f01fc0a9750 2024-03-29T11:01:39.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.684+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01f0001b90 con 0x7f01fc0a9750 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.684+0000 7f01e37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.684+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f01f0003a60 con 0x7f01fc0a9750 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.684+0000 7f01e37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.684+0000 7f01e37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.684+0000 7f01e37fe640 10 monclient: dump: 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:39.665 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:39.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.685+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f01f0003c80 con 0x7f01fc0a9750 2024-03-29T11:01:39.666 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.686+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f01f0084af0 con 0x7f01fc0a9750 2024-03-29T11:01:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.686+0000 7f01e37fe640 10 client.4583.objecter ms_dispatch 0x7f01fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.686+0000 7f01e37fe640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.686+0000 7f01e37fe640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.686+0000 7f01f8ff9640 4 set_mon_vals no callback set 2024-03-29T11:01:39.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.686+0000 7f01e37fe640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:01:39.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.689+0000 7f01fa7fc640 10 monclient: get_auth_request con 0x7f01e806dbd0 auth_method 0 2024-03-29T11:01:39.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.690+0000 7f01e37fe640 10 client.4583.objecter ms_handle_connect 0x7f01e806dbd0 2024-03-29T11:01:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.695+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01f0050430 con 0x7f01fc0a9750 2024-03-29T11:01:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.695+0000 7f01e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:39.674 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:39.695+0000 7f01e37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:40.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.089+0000 7f01e17fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:40.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.089+0000 7f01e17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:40.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.089+0000 7f01e17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:01:40.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.089+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01fc06a370 con 0x7f01fc0a9750 2024-03-29T11:01:40.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.091+0000 7f01e37fe640 1 -- 172.21.15.22:0/246310263 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f01f0052360 con 0x7f01fc0a9750 2024-03-29T11:01:40.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.091+0000 7f01e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:40.070 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.091+0000 7f01e37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:40.071 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:40.071 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:40.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.094+0000 7f01e17fa640 10 librados: watch_flush enter 2024-03-29T11:01:40.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.094+0000 7f01e17fa640 10 librados: watch_flush exit 2024-03-29T11:01:40.074 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.094+0000 7f01e17fa640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:01:40.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.094+0000 7f01e17fa640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:01:40.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.095+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 >> v1:172.21.15.22:6816/380221644 conn(0x7f01e806dbd0 legacy=0x7f01e8070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:40.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.095+0000 7f01e17fa640 10 monclient: shutdown 2024-03-29T11:01:40.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.095+0000 7f01e17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:40.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.095+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 >> v1:172.21.15.22:6790/0 conn(0x7f01fc0a9750 legacy=0x7f01fc071c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:40.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.095+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 shutdown_connections 2024-03-29T11:01:40.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.095+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 >> 172.21.15.22:0/246310263 conn(0x7f01fc0b3a30 msgr2=0x7f01fc0b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:40.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.096+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 shutdown_connections 2024-03-29T11:01:40.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.096+0000 7f01e17fa640 1 -- 172.21.15.22:0/246310263 wait complete. 2024-03-29T11:01:40.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:40.097+0000 7f01e17fa640 1 librados: shutdown 2024-03-29T11:01:40.096 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:46.901 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:47.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.065+0000 7f032f59e640 10 monclient: get_monmap_and_config 2024-03-29T11:01:47.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.065+0000 7f032f59e640 10 monclient: build_initial_monmap 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.065+0000 7f032f59e640 10 monclient: monmap: 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:47.066337+0000 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:47.066337+0000 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:47.046 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:47.048 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.066+0000 7f032f59e640 1 Processor -- start 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.066+0000 7f032f59e640 1 -- start start 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.066+0000 7f032f59e640 10 monclient: init 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f033014a400 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f03300aa500 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0330149800 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03300aca50 con 0x7f033014a400 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0330149cb0 con 0x7f0330149800 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03300acc80 con 0x7f03300aa500 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:47.049 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.067+0000 7f032f59e640 10 monclient(hunting): authenticate will time out at 1208.371848s 2024-03-29T11:01:47.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.070+0000 7f032dd9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f03300aa500 0x7f03300ad160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52578/0 (socket says 172.21.15.22:52578) 2024-03-29T11:01:47.050 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.070+0000 7f032dd9b640 1 -- 172.21.15.22:0/1034926108 learned_addr learned my addr 172.21.15.22:0/1034926108 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:47.055 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039077649 0 0) 0x7f03300acc80 con 0x7f03300aa500 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0318003520 con 0x7f03300aa500 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525733637 0 0) 0x7f0330149cb0 con 0x7f0330149800 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.072+0000 7f032d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03300acc80 con 0x7f0330149800 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778648044 0 0) 0x7f03300aca50 con 0x7f033014a400 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:01:47.056 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 10 monclient(hunting): my global_id is 4461 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0330149cb0 con 0x7f033014a400 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924913633 0 0) 0x7f0318003520 con 0x7f03300aa500 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 >> v1:172.21.15.26:6789/0 conn(0x7f0330149800 legacy=0x7f0330151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 >> v1:172.21.15.22:6789/0 conn(0x7f033014a400 legacy=0x7f03300aac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 monclient: found mon.noname-b 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.073+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03300ac350 con 0x7f03300aa500 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 10 monclient: _finish_auth 0 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:47.057 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032f59e640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0328001610 con 0x7f03300aa500 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 10 monclient: dump: 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:47.058 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032f59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0328001830 con 0x7f03300aa500 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032f59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032f59e640 4 set_mon_vals no callback set 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 1 -- 172.21.15.22:0/1034926108 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0328001c00 con 0x7f03300aa500 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.074+0000 7f032d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.075+0000 7f032d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.075+0000 7f032d59a640 10 monclient: dump: 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.059 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.075+0000 7f032f59e640 10 monclient: shutdown 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.075+0000 7f032f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:47.060 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.075+0000 7f032f59e640 1 -- 172.21.15.22:0/1034926108 >> v1:172.21.15.22:6790/0 conn(0x7f03300aa500 legacy=0x7f03300ad160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.078+0000 7f032f59e640 1 -- 172.21.15.22:0/1034926108 shutdown_connections 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.078+0000 7f032f59e640 1 -- 172.21.15.22:0/1034926108 >> 172.21.15.22:0/1034926108 conn(0x7f03300b3a30 msgr2=0x7f03300b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.079+0000 7f032f59e640 1 -- 172.21.15.22:0/1034926108 shutdown_connections 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.080+0000 7f032f59e640 1 -- 172.21.15.22:0/1034926108 wait complete. 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.081+0000 7f032f59e640 10 monclient: build_initial_monmap 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.081+0000 7f032f59e640 10 monclient: monmap: 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:47.061 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.081+0000 7f032f59e640 1 librados: starting msgr at 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.081+0000 7f032f59e640 1 librados: starting objecter 2024-03-29T11:01:47.062 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.082+0000 7f032f59e640 1 Processor -- start 2024-03-29T11:01:47.063 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.082+0000 7f032f59e640 1 -- start start 2024-03-29T11:01:47.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.082+0000 7f032f59e640 1 librados: setting wanted keys 2024-03-29T11:01:47.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.082+0000 7f032f59e640 1 librados: calling monclient init 2024-03-29T11:01:47.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.082+0000 7f032f59e640 10 monclient: init 2024-03-29T11:01:47.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.084+0000 7f032f59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:47.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.084+0000 7f032f59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:47.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.084+0000 7f032f59e640 10 monclient(hunting): picked mon.noname-a con 0x7f03300aa500 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:47.064 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.084+0000 7f032f59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0330149800 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:47.065 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.086+0000 7f032dd9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0330149800 0x7f0330137120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:53204/0 (socket says 172.21.15.22:53204) 2024-03-29T11:01:47.066 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.086+0000 7f032f59e640 10 monclient(hunting): picked mon.noname-b con 0x7f033014a400 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:47.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.087+0000 7f032f59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03301387e0 con 0x7f03300aa500 2024-03-29T11:01:47.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.087+0000 7f032f59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0330147010 con 0x7f033014a400 2024-03-29T11:01:47.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.087+0000 7f032f59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f033013ef90 con 0x7f0330149800 2024-03-29T11:01:47.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.087+0000 7f032f59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:47.067 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.087+0000 7f032f59e640 10 monclient(hunting): authenticate will time out at 1208.391634s 2024-03-29T11:01:47.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.087+0000 7f032dd9b640 1 -- 172.21.15.22:0/3757580611 learned_addr learned my addr 172.21.15.22:0/3757580611 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:47.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 10 client.?.objecter ms_handle_connect 0x7f0330149800 2024-03-29T11:01:47.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:47.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 238641504 0 0) 0x7f033013ef90 con 0x7f0330149800 2024-03-29T11:01:47.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:47.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:47.068 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:47.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:01:47.069 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.088+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0310004eb0 con 0x7f0330149800 2024-03-29T11:01:47.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 10 client.?.objecter ms_handle_connect 0x7f03300aa500 2024-03-29T11:01:47.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:47.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757684342 0 0) 0x7f03301387e0 con 0x7f03300aa500 2024-03-29T11:01:47.075 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 10 monclient(hunting): _negotiate 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.090+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f033013ef90 con 0x7f03300aa500 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 176903600 0 0) 0x7f033013ef90 con 0x7f03300aa500 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03301387e0 con 0x7f03300aa500 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0320001740 con 0x7f03300aa500 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 10 monclient(hunting): dump: 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.076 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 10 client.?.objecter ms_handle_connect 0x7f033014a400 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1761713361 0 0) 0x7f03301387e0 con 0x7f03300aa500 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 >> v1:172.21.15.22:6790/0 conn(0x7f0330149800 legacy=0x7f0330137120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 >> v1:172.21.15.26:6789/0 conn(0x7f033014a400 legacy=0x7f0330138c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 10 monclient: _finish_hunting 0 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.091+0000 7f031effd640 1 monclient: found mon.a 2024-03-29T11:01:47.077 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0330144f70 con 0x7f03300aa500 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 10 monclient: _finish_auth 0 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 10 monclient: _check_auth_tickets 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 10 monclient: _renew_subs 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0330137830 con 0x7f03300aa500 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0320001c50 con 0x7f03300aa500 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f03200037b0 con 0x7f03300aa500 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:47.078 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f031effd640 10 monclient: dump: 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032cd99640 4 set_mon_vals no callback set 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 10 client.4464.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 10 monclient: _renew_subs 2024-03-29T11:01:47.079 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:47.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0330139380 con 0x7f03300aa500 2024-03-29T11:01:47.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.092+0000 7f032f59e640 1 librados: init done 2024-03-29T11:01:47.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.094+0000 7f032f59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:47.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.094+0000 7f032f59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:47.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.094+0000 7f032f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:47.080 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.094+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03301451c0 con 0x7f03300aa500 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.096+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f0320003910 con 0x7f03300aa500 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.097+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0320083750 con 0x7f03300aa500 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.097+0000 7f031effd640 10 client.4464.objecter ms_dispatch 0x7f0330000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.098+0000 7f032dd9b640 10 monclient: get_auth_request con 0x7f031006dd90 auth_method 0 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.098+0000 7f031effd640 3 client.4464.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.098+0000 7f031effd640 3 client.4464.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.098+0000 7f031effd640 20 client.4464.objecter dump_active .. 0 homeless 2024-03-29T11:01:47.081 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.098+0000 7f031effd640 10 client.4464.objecter ms_handle_connect 0x7f031006dd90 2024-03-29T11:01:47.086 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.104+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f032004f030 con 0x7f03300aa500 2024-03-29T11:01:47.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.105+0000 7f031effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:47.087 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.105+0000 7f031effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:47.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.483+0000 7f032f59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:47.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.483+0000 7f032f59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:47.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.483+0000 7f032f59e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:47.463 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.483+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0330146430 con 0x7f03300aa500 2024-03-29T11:01:47.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.484+0000 7f031effd640 1 -- 172.21.15.22:0/3757580611 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0320050f60 con 0x7f03300aa500 2024-03-29T11:01:47.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.484+0000 7f031effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:47.464 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.484+0000 7f031effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:47.465 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:47.465 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:47.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 10 librados: watch_flush enter 2024-03-29T11:01:47.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 10 librados: watch_flush exit 2024-03-29T11:01:47.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 20 client.4464.objecter shutdown clearing up homeless session... 2024-03-29T11:01:47.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 10 client.4464.objecter successfully canceled tick 2024-03-29T11:01:47.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 >> v1:172.21.15.22:6816/380221644 conn(0x7f031006dd90 legacy=0x7f0310070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:47.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 10 monclient: shutdown 2024-03-29T11:01:47.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:47.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.488+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 >> v1:172.21.15.22:6789/0 conn(0x7f03300aa500 legacy=0x7f0330145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:47.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.489+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 shutdown_connections 2024-03-29T11:01:47.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.489+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 >> 172.21.15.22:0/3757580611 conn(0x7f03300b3a30 msgr2=0x7f0330069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:47.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.489+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 shutdown_connections 2024-03-29T11:01:47.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.489+0000 7f032f59e640 1 -- 172.21.15.22:0/3757580611 wait complete. 2024-03-29T11:01:47.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:47.489+0000 7f032f59e640 1 librados: shutdown 2024-03-29T11:01:47.488 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:01:54.295 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:01:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.453+0000 7f6a67597640 10 monclient: get_monmap_and_config 2024-03-29T11:01:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.453+0000 7f6a67597640 10 monclient: build_initial_monmap 2024-03-29T11:01:54.433 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.453+0000 7f6a67597640 10 monclient: monmap: 2024-03-29T11:01:54.433 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:54.433 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:54.433 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:01:54.454401+0000 2024-03-29T11:01:54.433 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:01:54.454401+0000 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.454+0000 7f6a67597640 1 Processor -- start 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.454+0000 7f6a67597640 1 -- start start 2024-03-29T11:01:54.434 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.454+0000 7f6a67597640 10 monclient: init 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 10 monclient(hunting): picked mon.noname-c con 0x7f6a600b4330 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 10 monclient(hunting): picked mon.noname-a con 0x7f6a601470b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 10 monclient(hunting): picked mon.noname-b con 0x7f6a601439b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6006f2a0 con 0x7f6a601470b0 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a60146b90 con 0x7f6a600b4330 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a600b2ca0 con 0x7f6a601439b0 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:54.438 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a67597640 10 monclient(hunting): authenticate will time out at 1215.760491s 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a65b34640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6a601439b0 0x7f6a60151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:48442/0 (socket says 172.21.15.22:48442) 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.456+0000 7f6a65b34640 1 -- 172.21.15.22:0/474881808 learned_addr learned my addr 172.21.15.22:0/474881808 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626602609 0 0) 0x7f6a600b2ca0 con 0x7f6a601439b0 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 10 monclient(hunting): _negotiate 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a4c003520 con 0x7f6a601439b0 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1435504009 0 0) 0x7f6a4c003520 con 0x7f6a601439b0 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.457+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 >> v1:172.21.15.26:6789/0 conn(0x7f6a600b4330 legacy=0x7f6a600b4710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:01:54.439 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 >> v1:172.21.15.22:6789/0 conn(0x7f6a601470b0 legacy=0x7f6a600b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 10 monclient: _finish_hunting 0 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 1 monclient: found mon.noname-b 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6006e0c0 con 0x7f6a601439b0 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 10 monclient: _finish_auth 0 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 10 monclient: _check_auth_tickets 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6a58001690 con 0x7f6a601439b0 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.458+0000 7f6a57fff640 10 monclient: dump: 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:54.440 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.441 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.441 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:54.441 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:54.441 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:54.441 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:54.441 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:54.441 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:54.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a67597640 5 monclient: authenticate success, global_id 4592 2024-03-29T11:01:54.442 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a67597640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a58001860 con 0x7f6a601439b0 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a57fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a57fff640 1 -- 172.21.15.22:0/474881808 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6a58001c70 con 0x7f6a601439b0 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a57fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a57fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a57fff640 10 monclient: dump: 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:54.443 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a67597640 10 monclient: get_monmap_and_config success 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.459+0000 7f6a67597640 4 set_mon_vals no callback set 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.460+0000 7f6a67597640 10 monclient: shutdown 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.460+0000 7f6a67597640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.460+0000 7f6a67597640 1 -- 172.21.15.22:0/474881808 >> v1:172.21.15.22:6790/0 conn(0x7f6a601439b0 legacy=0x7f6a60151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.461+0000 7f6a67597640 1 -- 172.21.15.22:0/474881808 shutdown_connections 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.461+0000 7f6a67597640 1 -- 172.21.15.22:0/474881808 >> 172.21.15.22:0/474881808 conn(0x7f6a60066a00 msgr2=0x7f6a60066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.461+0000 7f6a67597640 1 -- 172.21.15.22:0/474881808 shutdown_connections 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.461+0000 7f6a67597640 1 -- 172.21.15.22:0/474881808 wait complete. 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.462+0000 7f6a67597640 10 monclient: build_initial_monmap 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.462+0000 7f6a67597640 10 monclient: monmap: 2024-03-29T11:01:54.444 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.462+0000 7f6a67597640 1 librados: starting msgr at 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.462+0000 7f6a67597640 1 librados: starting objecter 2024-03-29T11:01:54.445 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.462+0000 7f6a67597640 1 Processor -- start 2024-03-29T11:01:54.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.463+0000 7f6a67597640 1 -- start start 2024-03-29T11:01:54.452 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.463+0000 7f6a67597640 1 librados: setting wanted keys 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.463+0000 7f6a67597640 1 librados: calling monclient init 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.463+0000 7f6a67597640 10 monclient: init 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 10 monclient: _reopen_session rank -1 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 10 monclient(hunting): picked mon.noname-a con 0x7f6a600b4330 addr v1:172.21.15.22:6789/0 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 10 monclient(hunting): picked mon.noname-c con 0x7f6a601439b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 10 monclient(hunting): picked mon.noname-b con 0x7f6a601470b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a601df580 con 0x7f6a600b4330 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a601df2c0 con 0x7f6a601470b0 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6011f770 con 0x7f6a601439b0 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 10 monclient(hunting): _renew_subs 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.464+0000 7f6a67597640 10 monclient(hunting): authenticate will time out at 1215.769080s 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.466+0000 7f6a64b32640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6a601439b0 0x7f6a6011fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:48894/0 (socket says 172.21.15.22:48894) 2024-03-29T11:01:54.453 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.466+0000 7f6a64b32640 1 -- 172.21.15.22:0/58388507 learned_addr learned my addr 172.21.15.22:0/58388507 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.466+0000 7f6a55ffb640 10 client.?.objecter ms_handle_connect 0x7f6a601439b0 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.466+0000 7f6a55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3071043130 0 0) 0x7f6a6011f770 con 0x7f6a601439b0 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a44004eb0 con 0x7f6a601439b0 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 client.?.objecter ms_handle_connect 0x7f6a600b4330 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893282306 0 0) 0x7f6a601df580 con 0x7f6a600b4330 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:01:54.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a44006230 con 0x7f6a600b4330 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2484429282 0 0) 0x7f6a44006230 con 0x7f6a600b4330 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a601df580 con 0x7f6a600b4330 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6a5c001b10 con 0x7f6a600b4330 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.467+0000 7f6a55ffb640 10 monclient(hunting): dump: 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:54.455 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.468+0000 7f6a55ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.468+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3553515567 0 0) 0x7f6a601df580 con 0x7f6a600b4330 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.468+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 >> v1:172.21.15.22:6790/0 conn(0x7f6a601439b0 legacy=0x7f6a6011fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.468+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 >> v1:172.21.15.26:6789/0 conn(0x7f6a601470b0 legacy=0x7f6a601205b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 1 monclient: found mon.a 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60142ce0 con 0x7f6a600b4330 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 10 monclient: _finish_auth 0 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:01:54.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 5 monclient: authenticate success, global_id 4470 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 10 monclient: _renew_subs 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a60142f30 con 0x7f6a600b4330 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 10 monclient: _renew_subs 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a601e70e0 con 0x7f6a600b4330 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.469+0000 7f6a67597640 1 librados: init done 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a5c002020 con 0x7f6a600b4330 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6a5c003b10 con 0x7f6a600b4330 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a55ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:01:54.457 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a55ffb640 10 monclient: dump: 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a67597640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a67597640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a67597640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.471+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6006fca0 con 0x7f6a600b4330 2024-03-29T11:01:54.458 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.472+0000 7f6a577fe640 4 set_mon_vals no callback set 2024-03-29T11:01:54.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.478+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f6a5c019ae0 con 0x7f6a600b4330 2024-03-29T11:01:54.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.478+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f6a5c083a60 con 0x7f6a600b4330 2024-03-29T11:01:54.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.478+0000 7f6a55ffb640 10 client.4470.objecter ms_dispatch 0x7f6a60000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:01:54.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.478+0000 7f6a55ffb640 3 client.4470.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:01:54.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.478+0000 7f6a55ffb640 3 client.4470.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:01:54.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.478+0000 7f6a55ffb640 20 client.4470.objecter dump_active .. 0 homeless 2024-03-29T11:01:54.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.479+0000 7f6a64b32640 10 monclient: get_auth_request con 0x7f6a4406dd60 auth_method 0 2024-03-29T11:01:54.460 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.479+0000 7f6a55ffb640 10 client.4470.objecter ms_handle_connect 0x7f6a4406dd60 2024-03-29T11:01:54.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.480+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a5c04f170 con 0x7f6a600b4330 2024-03-29T11:01:54.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.480+0000 7f6a55ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:01:54.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.480+0000 7f6a55ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:01:54.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.816+0000 7f6a67597640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:54.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.816+0000 7f6a67597640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:54.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.816+0000 7f6a67597640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:01:54.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.816+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a60074f10 con 0x7f6a600b4330 2024-03-29T11:01:54.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.817+0000 7f6a55ffb640 1 -- 172.21.15.22:0/58388507 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f6a5c051270 con 0x7f6a600b4330 2024-03-29T11:01:54.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.817+0000 7f6a55ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:01:54.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.817+0000 7f6a55ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:01:54.799 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:01:54.799 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 10 librados: watch_flush enter 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 10 librados: watch_flush exit 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 20 client.4470.objecter shutdown clearing up homeless session... 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 10 client.4470.objecter successfully canceled tick 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 >> v1:172.21.15.22:6816/380221644 conn(0x7f6a4406dd60 legacy=0x7f6a44070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 10 monclient: shutdown 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.822+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 >> v1:172.21.15.22:6789/0 conn(0x7f6a600b4330 legacy=0x7f6a601debb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:01:54.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.823+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 shutdown_connections 2024-03-29T11:01:54.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.823+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 >> 172.21.15.22:0/58388507 conn(0x7f6a60066a00 msgr2=0x7f6a600704a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:01:54.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.823+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 shutdown_connections 2024-03-29T11:01:54.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.823+0000 7f6a67597640 1 -- 172.21.15.22:0/58388507 wait complete. 2024-03-29T11:01:54.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:01:54.823+0000 7f6a67597640 1 librados: shutdown 2024-03-29T11:01:54.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:01.627 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.799+0000 7fc74c36b640 10 monclient: get_monmap_and_config 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.799+0000 7fc74c36b640 10 monclient: build_initial_monmap 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.799+0000 7fc74c36b640 10 monclient: monmap: 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:01.800311+0000 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:01.800311+0000 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:01.779 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:01.780 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:01.780 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:01.780 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:01.780 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:01.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.800+0000 7fc74c36b640 1 Processor -- start 2024-03-29T11:02:01.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.800+0000 7fc74c36b640 1 -- start start 2024-03-29T11:02:01.780 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.800+0000 7fc74c36b640 10 monclient: init 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.801+0000 7fc74c36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.801+0000 7fc74c36b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74a107640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc74406d9d0 0x7fc7440b05f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40032/0 (socket says 172.21.15.22:40032) 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74a107640 1 -- 172.21.15.22:0/2313215562 learned_addr learned my addr 172.21.15.22:0/2313215562 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 10 monclient(hunting): picked mon.noname-b con 0x7fc74406d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 10 monclient(hunting): picked mon.noname-c con 0x7fc7440af1d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 10 monclient(hunting): picked mon.noname-a con 0x7fc744147a60 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7440aed10 con 0x7fc744147a60 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7440afd10 con 0x7fc7440af1d0 2024-03-29T11:02:01.785 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7440ab490 con 0x7fc74406d9d0 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc74c36b640 10 monclient(hunting): authenticate will time out at 1223.107053s 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.802+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090867251 0 0) 0x7fc7440afd10 con 0x7fc7440af1d0 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): _negotiate 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc730003440 con 0x7fc7440af1d0 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119618692 0 0) 0x7fc7440ab490 con 0x7fc74406d9d0 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): _negotiate 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:01.786 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7440afd10 con 0x7fc74406d9d0 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 513667677 0 0) 0x7fc730003440 con 0x7fc7440af1d0 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 >> v1:172.21.15.22:6790/0 conn(0x7fc74406d9d0 legacy=0x7fc7440b05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 >> v1:172.21.15.22:6789/0 conn(0x7fc744147a60 legacy=0x7fc744149e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: _finish_hunting 0 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 monclient: found mon.noname-c 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7440ab770 con 0x7fc7440af1d0 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: _finish_auth 0 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: _check_auth_tickets 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc734001690 con 0x7fc7440af1d0 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:01.787 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: dump: 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc74c36b640 5 monclient: authenticate success, global_id 4447 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc74c36b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc734001870 con 0x7fc7440af1d0 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:01.788 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 1 -- 172.21.15.22:0/2313215562 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc734001690 con 0x7fc7440af1d0 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc749105640 10 monclient: dump: 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc74c36b640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:01.789 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.803+0000 7fc74c36b640 4 set_mon_vals no callback set 2024-03-29T11:02:01.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.804+0000 7fc74c36b640 10 monclient: shutdown 2024-03-29T11:02:01.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.804+0000 7fc74c36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:01.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.804+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 >> v1:172.21.15.26:6789/0 conn(0x7fc7440af1d0 legacy=0x7fc7440af600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:01.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.804+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 shutdown_connections 2024-03-29T11:02:01.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.804+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 >> 172.21.15.22:0/2313215562 conn(0x7fc74406b660 msgr2=0x7fc74406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:01.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.805+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 shutdown_connections 2024-03-29T11:02:01.790 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.805+0000 7fc74c36b640 1 -- 172.21.15.22:0/2313215562 wait complete. 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 10 monclient: build_initial_monmap 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 10 monclient: monmap: 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:01.794 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 1 librados: starting msgr at 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 1 librados: starting objecter 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 1 Processor -- start 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 1 -- start start 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 1 librados: setting wanted keys 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 1 librados: calling monclient init 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.806+0000 7fc74c36b640 10 monclient: init 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.807+0000 7fc74c36b640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:01.795 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.807+0000 7fc74c36b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.807+0000 7fc74c36b640 10 monclient(hunting): picked mon.noname-c con 0x7fc74406d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.807+0000 7fc74c36b640 10 monclient(hunting): picked mon.noname-b con 0x7fc7440af1d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.807+0000 7fc74a107640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fc74406d9d0 0x7fc7440b3c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40230/0 (socket says 172.21.15.22:40230) 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.807+0000 7fc74a107640 1 -- 172.21.15.22:0/382050393 learned_addr learned my addr 172.21.15.22:0/382050393 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 client.?.objecter ms_handle_connect 0x7fc74406d9d0 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 client.?.objecter ms_handle_connect 0x7fc744147a60 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc74c36b640 10 monclient(hunting): picked mon.noname-a con 0x7fc744147a60 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74406e1b0 con 0x7fc744147a60 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 client.?.objecter ms_handle_connect 0x7fc7440af1d0 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74406e640 con 0x7fc7440af1d0 2024-03-29T11:02:01.796 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc74406f920 con 0x7fc74406d9d0 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc74c36b640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc74c36b640 10 monclient(hunting): authenticate will time out at 1223.112771s 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776329033 0 0) 0x7fc74406e640 con 0x7fc7440af1d0 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.808+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc728004eb0 con 0x7fc7440af1d0 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606548622 0 0) 0x7fc74406f920 con 0x7fc74406d9d0 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc74406e640 con 0x7fc74406d9d0 2024-03-29T11:02:01.797 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 675307996 0 0) 0x7fc728004eb0 con 0x7fc7440af1d0 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc74406f920 con 0x7fc7440af1d0 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc734001690 con 0x7fc7440af1d0 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): dump: 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.798 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652232355 0 0) 0x7fc74406e1b0 con 0x7fc744147a60 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7280057f0 con 0x7fc744147a60 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.809+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3088008361 0 0) 0x7fc74406e640 con 0x7fc74406d9d0 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc74406e1b0 con 0x7fc74406d9d0 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc740001910 con 0x7fc74406d9d0 2024-03-29T11:02:01.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient(hunting): dump: 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2772747660 0 0) 0x7fc74406f920 con 0x7fc7440af1d0 2024-03-29T11:02:01.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 >> v1:172.21.15.22:6790/0 conn(0x7fc74406d9d0 legacy=0x7fc7440b3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 >> v1:172.21.15.22:6789/0 conn(0x7fc744147a60 legacy=0x7fc7441e6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 monclient: found mon.b 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc74406fc00 con 0x7fc7440af1d0 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient: _finish_auth 0 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 10 monclient: _renew_subs 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc74406ddb0 con 0x7fc7440af1d0 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:01.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 10 monclient: _renew_subs 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc74406ea30 con 0x7fc7440af1d0 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc74c36b640 1 librados: init done 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc734003970 con 0x7fc7440af1d0 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.810+0000 7fc73affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.811+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fc734003c80 con 0x7fc7440af1d0 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.811+0000 7fc73affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.811+0000 7fc73affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.811+0000 7fc73affd640 10 monclient: dump: 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:01.802 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.811+0000 7fc748904640 4 set_mon_vals no callback set 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.812+0000 7fc74c36b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.812+0000 7fc74c36b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.812+0000 7fc74c36b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.812+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc72c005080 con 0x7fc7440af1d0 2024-03-29T11:02:01.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.814+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fc734019c50 con 0x7fc7440af1d0 2024-03-29T11:02:01.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.815+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fc734083720 con 0x7fc7440af1d0 2024-03-29T11:02:01.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.815+0000 7fc73affd640 10 client.4450.objecter ms_dispatch 0x7fc744000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:01.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.815+0000 7fc74a107640 10 monclient: get_auth_request con 0x7fc72806e300 auth_method 0 2024-03-29T11:02:01.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.815+0000 7fc73affd640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:01.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.815+0000 7fc73affd640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:01.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.815+0000 7fc73affd640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T11:02:01.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.816+0000 7fc73affd640 10 client.4450.objecter ms_handle_connect 0x7fc72806e300 2024-03-29T11:02:01.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.822+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc73404ef70 con 0x7fc7440af1d0 2024-03-29T11:02:01.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.822+0000 7fc73affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:01.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:01.822+0000 7fc73affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:02.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.235+0000 7fc74c36b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:02.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.235+0000 7fc74c36b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:02.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.235+0000 7fc74c36b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:02.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.235+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc72c005330 con 0x7fc7440af1d0 2024-03-29T11:02:02.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.236+0000 7fc73affd640 1 -- 172.21.15.22:0/382050393 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fc73404ef70 con 0x7fc7440af1d0 2024-03-29T11:02:02.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.236+0000 7fc73affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:02.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.236+0000 7fc73affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:02.216 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:02.216 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:02.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 10 librados: watch_flush enter 2024-03-29T11:02:02.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 10 librados: watch_flush exit 2024-03-29T11:02:02.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T11:02:02.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 10 client.4450.objecter successfully canceled tick 2024-03-29T11:02:02.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 >> v1:172.21.15.22:6816/380221644 conn(0x7fc72806e300 legacy=0x7fc7280707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:02.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 10 monclient: shutdown 2024-03-29T11:02:02.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:02.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.239+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 >> v1:172.21.15.26:6789/0 conn(0x7fc7440af1d0 legacy=0x7fc7441dee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:02.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.240+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 shutdown_connections 2024-03-29T11:02:02.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.240+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 >> 172.21.15.22:0/382050393 conn(0x7fc74406b660 msgr2=0x7fc744146e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:02.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.240+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 shutdown_connections 2024-03-29T11:02:02.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.240+0000 7fc74c36b640 1 -- 172.21.15.22:0/382050393 wait complete. 2024-03-29T11:02:02.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:02.241+0000 7fc74c36b640 1 librados: shutdown 2024-03-29T11:02:02.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:09.042 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:09.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.203+0000 7f839540d640 10 monclient: get_monmap_and_config 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.203+0000 7f839540d640 10 monclient: build_initial_monmap 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.203+0000 7f839540d640 10 monclient: monmap: 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:09.203853+0000 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:09.203853+0000 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:09.183 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:09.184 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:09.184 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.204+0000 7f839540d640 1 Processor -- start 2024-03-29T11:02:09.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.204+0000 7f839540d640 1 -- start start 2024-03-29T11:02:09.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.204+0000 7f839540d640 10 monclient: init 2024-03-29T11:02:09.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.205+0000 7f839540d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:09.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.205+0000 7f839540d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:09.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.205+0000 7f839540d640 10 monclient(hunting): picked mon.noname-b con 0x7f83901457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:09.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.205+0000 7f839540d640 10 monclient(hunting): picked mon.noname-c con 0x7f839013f650 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:09.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.205+0000 7f838effd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f83901457f0 0x7f839013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37202/0 (socket says 172.21.15.22:37202) 2024-03-29T11:02:09.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f839540d640 10 monclient(hunting): picked mon.noname-a con 0x7f83900a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:09.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838effd640 1 -- 172.21.15.22:0/4284268210 learned_addr learned my addr 172.21.15.22:0/4284268210 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:09.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83900aaef0 con 0x7f83900a9d70 2024-03-29T11:02:09.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8390145bd0 con 0x7f839013f650 2024-03-29T11:02:09.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8390145eb0 con 0x7f83901457f0 2024-03-29T11:02:09.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f839540d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f839540d640 10 monclient(hunting): authenticate will time out at 1230.511293s 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621578566 0 0) 0x7f8390145eb0 con 0x7f83901457f0 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f837c003520 con 0x7f83901457f0 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2065006967 0 0) 0x7f837c003520 con 0x7f83901457f0 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.207+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 >> v1:172.21.15.26:6789/0 conn(0x7f839013f650 legacy=0x7f83900b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:09.189 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.208+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 >> v1:172.21.15.22:6789/0 conn(0x7f83900a9d70 legacy=0x7f839014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:09.193 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390146340 con 0x7f83901457f0 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f8380001690 con 0x7f83901457f0 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f838e7fc640 10 monclient: dump: 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:09.194 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f839540d640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.213+0000 7f839540d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.214+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8380001860 con 0x7f83901457f0 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.214+0000 7f838e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.214+0000 7f838e7fc640 1 -- 172.21.15.22:0/4284268210 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f8380001cb0 con 0x7f83901457f0 2024-03-29T11:02:09.195 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.214+0000 7f838e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.214+0000 7f838e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.214+0000 7f838e7fc640 10 monclient: dump: 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:09.196 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:09.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.216+0000 7f839540d640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:09.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.216+0000 7f839540d640 4 set_mon_vals no callback set 2024-03-29T11:02:09.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.217+0000 7f839540d640 10 monclient: shutdown 2024-03-29T11:02:09.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.217+0000 7f839540d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:09.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.217+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 >> v1:172.21.15.22:6790/0 conn(0x7f83901457f0 legacy=0x7f839013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:09.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.217+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 shutdown_connections 2024-03-29T11:02:09.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.217+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 >> 172.21.15.22:0/4284268210 conn(0x7f8390066a00 msgr2=0x7f8390066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:09.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.217+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 shutdown_connections 2024-03-29T11:02:09.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.217+0000 7f839540d640 1 -- 172.21.15.22:0/4284268210 wait complete. 2024-03-29T11:02:09.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.218+0000 7f839540d640 10 monclient: build_initial_monmap 2024-03-29T11:02:09.198 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.219+0000 7f839540d640 10 monclient: monmap: 2024-03-29T11:02:09.198 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:09.199 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:09.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.219+0000 7f839540d640 1 librados: starting msgr at 2024-03-29T11:02:09.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.219+0000 7f839540d640 1 librados: starting objecter 2024-03-29T11:02:09.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.220+0000 7f839540d640 1 Processor -- start 2024-03-29T11:02:09.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.220+0000 7f839540d640 1 -- start start 2024-03-29T11:02:09.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.220+0000 7f839540d640 1 librados: setting wanted keys 2024-03-29T11:02:09.201 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.220+0000 7f839540d640 1 librados: calling monclient init 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.220+0000 7f839540d640 10 monclient: init 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 10 monclient(hunting): picked mon.noname-a con 0x7f83900a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 10 monclient(hunting): picked mon.noname-b con 0x7f839013f650 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 10 monclient(hunting): picked mon.noname-c con 0x7f83901457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8390142250 con 0x7f83900a9d70 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8390142530 con 0x7f839013f650 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8390133570 con 0x7f83901457f0 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.221+0000 7f839540d640 10 monclient(hunting): authenticate will time out at 1230.525999s 2024-03-29T11:02:09.202 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.222+0000 7f838f7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f83901457f0 0x7f83901406f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:37402/0 (socket says 172.21.15.22:37402) 2024-03-29T11:02:09.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.222+0000 7f838f7fe640 1 -- 172.21.15.22:0/1976054999 learned_addr learned my addr 172.21.15.22:0/1976054999 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:09.204 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.223+0000 7f83877fe640 10 client.?.objecter ms_handle_connect 0x7f83901457f0 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.223+0000 7f83877fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.223+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896541276 0 0) 0x7f8390133570 con 0x7f83901457f0 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.223+0000 7f83877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.223+0000 7f83877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.223+0000 7f83877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.223+0000 7f83877fe640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.224+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8370004eb0 con 0x7f83901457f0 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.224+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3337253902 0 0) 0x7f8370004eb0 con 0x7f83901457f0 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.224+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8390133570 con 0x7f83901457f0 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.224+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f8388001b50 con 0x7f83901457f0 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.224+0000 7f83877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.224+0000 7f83877fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:09.205 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.224+0000 7f83877fe640 10 monclient(hunting): dump: 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.225+0000 7f83877fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.225+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1263545720 0 0) 0x7f8390133570 con 0x7f83901457f0 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.225+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 >> v1:172.21.15.26:6789/0 conn(0x7f839013f650 legacy=0x7f839013ffe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:09.206 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.225+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 >> v1:172.21.15.22:6789/0 conn(0x7f83900a9d70 legacy=0x7f8390132180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.228+0000 7f83877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.228+0000 7f83877fe640 1 monclient: found mon.c 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.228+0000 7f83877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.228+0000 7f83877fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390133850 con 0x7f83901457f0 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f839540d640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8388001cb0 con 0x7f83901457f0 2024-03-29T11:02:09.210 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f839540d640 10 monclient: _renew_subs 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f839540d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f839540d640 1 -- 172.21.15.22:0/1976054999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8390142cd0 con 0x7f83901457f0 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f8388004dc0 con 0x7f83901457f0 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f83877fe640 10 monclient: dump: 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:09.211 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f838d7fa640 4 set_mon_vals no callback set 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f839540d640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.229+0000 7f839540d640 10 monclient: _renew_subs 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.230+0000 7f839540d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.230+0000 7f839540d640 1 -- 172.21.15.22:0/1976054999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83901e3550 con 0x7f83901457f0 2024-03-29T11:02:09.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.230+0000 7f839540d640 1 librados: init done 2024-03-29T11:02:09.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.232+0000 7f83857fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:09.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.232+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f838801ad90 con 0x7f83901457f0 2024-03-29T11:02:09.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.232+0000 7f83857fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:09.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.232+0000 7f83857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:09.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.232+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83900b5410 con 0x7f83901457f0 2024-03-29T11:02:09.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.232+0000 7f838effd640 10 monclient: get_auth_request con 0x7f837006dcd0 auth_method 0 2024-03-29T11:02:09.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.233+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f8388084d10 con 0x7f83901457f0 2024-03-29T11:02:09.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.233+0000 7f83877fe640 10 client.4607.objecter ms_dispatch 0x7f8390000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:09.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.233+0000 7f83877fe640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:09.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.233+0000 7f83877fe640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:09.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.233+0000 7f83877fe640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-29T11:02:09.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.233+0000 7f83877fe640 10 client.4607.objecter ms_handle_connect 0x7f837006dcd0 2024-03-29T11:02:09.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.240+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8388050650 con 0x7f83901457f0 2024-03-29T11:02:09.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.240+0000 7f83877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:09.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.240+0000 7f83877fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:09.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.548+0000 7f83857fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:09.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.548+0000 7f83857fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:09.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.548+0000 7f83857fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:09.528 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.548+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83900aae40 con 0x7f83901457f0 2024-03-29T11:02:09.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.549+0000 7f83877fe640 1 -- 172.21.15.22:0/1976054999 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f8388052580 con 0x7f83901457f0 2024-03-29T11:02:09.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.549+0000 7f83877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:09.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.549+0000 7f83877fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:09.529 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:09.529 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:09.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 10 librados: watch_flush enter 2024-03-29T11:02:09.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 10 librados: watch_flush exit 2024-03-29T11:02:09.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-29T11:02:09.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 10 client.4607.objecter successfully canceled tick 2024-03-29T11:02:09.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 >> v1:172.21.15.22:6816/380221644 conn(0x7f837006dcd0 legacy=0x7f8370070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:09.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 10 monclient: shutdown 2024-03-29T11:02:09.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:09.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 >> v1:172.21.15.22:6790/0 conn(0x7f83901457f0 legacy=0x7f83901406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:09.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 shutdown_connections 2024-03-29T11:02:09.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.553+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 >> 172.21.15.22:0/1976054999 conn(0x7f8390066a00 msgr2=0x7f83900b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:09.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.554+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 shutdown_connections 2024-03-29T11:02:09.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.554+0000 7f83857fa640 1 -- 172.21.15.22:0/1976054999 wait complete. 2024-03-29T11:02:09.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:09.554+0000 7f83857fa640 1 librados: shutdown 2024-03-29T11:02:09.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:16.356 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:16.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.512+0000 7f7aa2510640 10 monclient: get_monmap_and_config 2024-03-29T11:02:16.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.512+0000 7f7aa2510640 10 monclient: build_initial_monmap 2024-03-29T11:02:16.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.512+0000 7f7aa2510640 10 monclient: monmap: 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:16.513242+0000 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:16.513242+0000 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:16.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.513+0000 7f7aa2510640 1 Processor -- start 2024-03-29T11:02:16.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.513+0000 7f7aa2510640 1 -- start start 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.513+0000 7f7aa2510640 10 monclient: init 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.514+0000 7f7aa2510640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.514+0000 7f7aa2510640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.514+0000 7f7aa2510640 10 monclient(hunting): picked mon.noname-b con 0x7f7a9c070170 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.514+0000 7f7aa2510640 10 monclient(hunting): picked mon.noname-a con 0x7f7a9c0713f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.514+0000 7f7aa2510640 10 monclient(hunting): picked mon.noname-c con 0x7f7a9c070550 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.514+0000 7f7aa2510640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c13dbd0 con 0x7f7a9c0713f0 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.515+0000 7f7aa2510640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c13ff70 con 0x7f7a9c070550 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.515+0000 7f7aa2510640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c140250 con 0x7f7a9c070170 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.515+0000 7f7aa2510640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:16.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.515+0000 7f7aa2510640 10 monclient(hunting): authenticate will time out at 1237.819299s 2024-03-29T11:02:16.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.515+0000 7f7a9bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7a9c070170 0x7f7a9c13d4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:60676/0 (socket says 172.21.15.22:60676) 2024-03-29T11:02:16.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.515+0000 7f7a9bfff640 1 -- 172.21.15.22:0/404290312 learned_addr learned my addr 172.21.15.22:0/404290312 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.517+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 695188148 0 0) 0x7f7a9c140250 con 0x7f7a9c070170 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a84003520 con 0x7f7a9c070170 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663688615 0 0) 0x7f7a9c13ff70 con 0x7f7a9c070550 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): _negotiate 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:02:16.504 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a9c140250 con 0x7f7a9c070550 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1945577688 0 0) 0x7f7a84003520 con 0x7f7a9c070170 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 >> v1:172.21.15.26:6789/0 conn(0x7f7a9c070550 legacy=0x7f7a9c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 >> v1:172.21.15.22:6789/0 conn(0x7f7a9c0713f0 legacy=0x7f7a9c1406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: _finish_hunting 0 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 monclient: found mon.noname-b 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c140df0 con 0x7f7a9c070170 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: _finish_auth 0 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: _check_auth_tickets 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7a94001a10 con 0x7f7a9c070170 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7aa2510640 5 monclient: authenticate success, global_id 4610 2024-03-29T11:02:16.505 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7aa2510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: dump: 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:16.506 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a94001c30 con 0x7f7a9c070170 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 1 -- 172.21.15.22:0/404290312 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7a94001fc0 con 0x7f7a9c070170 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.518+0000 7f7a9affd640 10 monclient: dump: 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:16.507 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.519+0000 7f7aa2510640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.519+0000 7f7aa2510640 4 set_mon_vals no callback set 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.520+0000 7f7aa2510640 10 monclient: shutdown 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.520+0000 7f7aa2510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.520+0000 7f7aa2510640 1 -- 172.21.15.22:0/404290312 >> v1:172.21.15.22:6790/0 conn(0x7f7a9c070170 legacy=0x7f7a9c13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.520+0000 7f7aa2510640 1 -- 172.21.15.22:0/404290312 shutdown_connections 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.520+0000 7f7aa2510640 1 -- 172.21.15.22:0/404290312 >> 172.21.15.22:0/404290312 conn(0x7f7a9c067610 msgr2=0x7f7a9c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.520+0000 7f7aa2510640 1 -- 172.21.15.22:0/404290312 shutdown_connections 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.520+0000 7f7aa2510640 1 -- 172.21.15.22:0/404290312 wait complete. 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.522+0000 7f7aa2510640 10 monclient: build_initial_monmap 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.522+0000 7f7aa2510640 10 monclient: monmap: 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:16.508 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.522+0000 7f7aa2510640 1 librados: starting msgr at 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.522+0000 7f7aa2510640 1 librados: starting objecter 2024-03-29T11:02:16.509 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.522+0000 7f7aa2510640 1 Processor -- start 2024-03-29T11:02:16.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.523+0000 7f7aa2510640 1 -- start start 2024-03-29T11:02:16.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.523+0000 7f7aa2510640 1 librados: setting wanted keys 2024-03-29T11:02:16.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.523+0000 7f7aa2510640 1 librados: calling monclient init 2024-03-29T11:02:16.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.523+0000 7f7aa2510640 10 monclient: init 2024-03-29T11:02:16.514 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 10 monclient(hunting): picked mon.noname-c con 0x7f7a9c070170 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 10 monclient(hunting): picked mon.noname-b con 0x7f7a9c070550 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7a9bfff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f7a9c070170 0x7f7a9c06bff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:32898/0 (socket says 172.21.15.22:32898) 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 10 monclient(hunting): picked mon.noname-a con 0x7f7a9c0713f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7a9bfff640 1 -- 172.21.15.22:0/1939257814 learned_addr learned my addr 172.21.15.22:0/1939257814 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c144310 con 0x7f7a9c0713f0 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c06d950 con 0x7f7a9c070550 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a9c06dc30 con 0x7f7a9c070170 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.524+0000 7f7aa2510640 10 monclient(hunting): authenticate will time out at 1237.829031s 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 client.?.objecter ms_handle_connect 0x7f7a9c070170 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:16.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952278121 0 0) 0x7f7a9c06dc30 con 0x7f7a9c070170 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): my global_id is 4613 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a88004eb0 con 0x7f7a9c070170 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 client.?.objecter ms_handle_connect 0x7f7a9c070550 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849244004 0 0) 0x7f7a9c06d950 con 0x7f7a9c070550 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a9c06dc30 con 0x7f7a9c070550 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.525+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1415548438 0 0) 0x7f7a88004eb0 con 0x7f7a9c070170 2024-03-29T11:02:16.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a9c06d950 con 0x7f7a9c070170 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7a94001f40 con 0x7f7a9c070170 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:16.517 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3602071545 0 0) 0x7f7a9c06dc30 con 0x7f7a9c070550 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a88004eb0 con 0x7f7a9c070550 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.526+0000 7f7a98ff9640 10 client.?.objecter ms_handle_connect 0x7f7a9c0713f0 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7a8c001830 con 0x7f7a9c070550 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient(hunting): dump: 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:16.518 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1729297745 0 0) 0x7f7a9c144310 con 0x7f7a9c0713f0 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:02:16.519 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a9c06dc30 con 0x7f7a9c0713f0 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1812265061 0 0) 0x7f7a9c06d950 con 0x7f7a9c070170 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 >> v1:172.21.15.26:6789/0 conn(0x7f7a9c070550 legacy=0x7f7a9c12caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 >> v1:172.21.15.22:6789/0 conn(0x7f7a9c0713f0 legacy=0x7f7a9c1e5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 monclient: found mon.c 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c06cce0 con 0x7f7a9c070170 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient: _finish_auth 0 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7aa2510640 5 monclient: authenticate success, global_id 4613 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7aa2510640 10 monclient: _renew_subs 2024-03-29T11:02:16.520 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7aa2510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a9c143e80 con 0x7f7a9c070170 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.527+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a940020a0 con 0x7f7a9c070170 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7aa2510640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7aa2510640 10 monclient: _renew_subs 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7aa2510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a9c1395b0 con 0x7f7a9c070170 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7aa2510640 1 librados: init done 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7a98ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7a94004fb0 con 0x7f7a9c070170 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7a98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7a98ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7a98ff9640 10 monclient: dump: 2024-03-29T11:02:16.521 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.528+0000 7f7a9a7fc640 4 set_mon_vals no callback set 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.529+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f7a9401af80 con 0x7f7a9c070170 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.529+0000 7f7aa2510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7aa2510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:16.522 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7aa2510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7c005080 con 0x7f7a9c070170 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f7a94084da0 con 0x7f7a9c070170 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7a98ff9640 10 client.4613.objecter ms_dispatch 0x7f7a9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7a98ff9640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7a98ff9640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.530+0000 7f7a98ff9640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.531+0000 7f7a9b7fe640 10 monclient: get_auth_request con 0x7f7a8806dfe0 auth_method 0 2024-03-29T11:02:16.523 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.531+0000 7f7a98ff9640 10 client.4613.objecter ms_handle_connect 0x7f7a8806dfe0 2024-03-29T11:02:16.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.538+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a94050680 con 0x7f7a9c070170 2024-03-29T11:02:16.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.538+0000 7f7a98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:16.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.538+0000 7f7a98ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:16.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.843+0000 7f7aa2510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:16.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.843+0000 7f7aa2510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:16.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.843+0000 7f7aa2510640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:16.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.843+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a7c005330 con 0x7f7a9c070170 2024-03-29T11:02:16.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.844+0000 7f7a98ff9640 1 -- 172.21.15.22:0/1939257814 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f7a940525b0 con 0x7f7a9c070170 2024-03-29T11:02:16.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.845+0000 7f7a98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:16.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.845+0000 7f7a98ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:16.825 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:16.825 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:16.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.848+0000 7f7aa2510640 10 librados: watch_flush enter 2024-03-29T11:02:16.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.848+0000 7f7aa2510640 10 librados: watch_flush exit 2024-03-29T11:02:16.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.848+0000 7f7aa2510640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-29T11:02:16.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.848+0000 7f7aa2510640 10 client.4613.objecter successfully canceled tick 2024-03-29T11:02:16.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.848+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 >> v1:172.21.15.22:6816/380221644 conn(0x7f7a8806dfe0 legacy=0x7f7a880704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:16.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 10 monclient: shutdown 2024-03-29T11:02:16.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:16.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 >> v1:172.21.15.22:6790/0 conn(0x7f7a9c070170 legacy=0x7f7a9c06bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:16.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 shutdown_connections 2024-03-29T11:02:16.829 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 >> 172.21.15.22:0/1939257814 conn(0x7f7a9c067610 msgr2=0x7f7a9c13d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:16.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 shutdown_connections 2024-03-29T11:02:16.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 1 -- 172.21.15.22:0/1939257814 wait complete. 2024-03-29T11:02:16.830 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:16.849+0000 7f7aa2510640 1 librados: shutdown 2024-03-29T11:02:16.843 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:23.651 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.818+0000 7f32af38c640 10 monclient: get_monmap_and_config 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.818+0000 7f32af38c640 10 monclient: build_initial_monmap 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.818+0000 7f32af38c640 10 monclient: monmap: 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:23.820463+0000 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:23.820463+0000 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:23.799 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:23.800 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:23.800 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:23.800 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:23.800 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:23.800 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.819+0000 7f32af38c640 1 Processor -- start 2024-03-29T11:02:23.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.819+0000 7f32af38c640 1 -- start start 2024-03-29T11:02:23.801 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.819+0000 7f32af38c640 10 monclient: init 2024-03-29T11:02:23.802 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 10 monclient(hunting): picked mon.noname-a con 0x7f32a81422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 10 monclient(hunting): picked mon.noname-b con 0x7f32a8142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 10 monclient(hunting): picked mon.noname-c con 0x7f32a80b17d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a8142c80 con 0x7f32a81422b0 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80aae40 con 0x7f32a80b17d0 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a813fbd0 con 0x7f32a8142690 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.820+0000 7f32af38c640 10 monclient(hunting): authenticate will time out at 1245.126039s 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.821+0000 7f32ac927640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f32a8142690 0x7f32a80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:39260/0 (socket says 172.21.15.22:39260) 2024-03-29T11:02:23.803 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.821+0000 7f32ac927640 1 -- 172.21.15.22:0/885769809 learned_addr learned my addr 172.21.15.22:0/885769809 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:23.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.823+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742922505 0 0) 0x7f32a813fbd0 con 0x7f32a8142690 2024-03-29T11:02:23.804 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.823+0000 7f329ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:23.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.823+0000 7f329ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:23.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.823+0000 7f329ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:23.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.823+0000 7f329ffff640 10 monclient(hunting): my global_id is 4616 2024-03-29T11:02:23.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.823+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3294003520 con 0x7f32a8142690 2024-03-29T11:02:23.805 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 213961560 0 0) 0x7f3294003520 con 0x7f32a8142690 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 >> v1:172.21.15.26:6789/0 conn(0x7f32a80b17d0 legacy=0x7f32a80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 >> v1:172.21.15.22:6789/0 conn(0x7f32a81422b0 legacy=0x7f32a813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 1 monclient: found mon.noname-b 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a8151940 con 0x7f32a8142690 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 10 monclient: _finish_auth 0 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f32a0001a10 con 0x7f32a8142690 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:23.806 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.824+0000 7f329ffff640 10 monclient: dump: 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.825+0000 7f32af38c640 5 monclient: authenticate success, global_id 4616 2024-03-29T11:02:23.807 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.825+0000 7f32af38c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:23.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.826+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a0001c30 con 0x7f32a8142690 2024-03-29T11:02:23.808 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.826+0000 7f329ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.826+0000 7f32af38c640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.826+0000 7f32af38c640 4 set_mon_vals no callback set 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f329ffff640 1 -- 172.21.15.22:0/885769809 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f32a0001fc0 con 0x7f32a8142690 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f329ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f329ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f329ffff640 10 monclient: dump: 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:23.809 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:23.810 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:23.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f32af38c640 10 monclient: shutdown 2024-03-29T11:02:23.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f32af38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:23.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f32af38c640 1 -- 172.21.15.22:0/885769809 >> v1:172.21.15.22:6790/0 conn(0x7f32a8142690 legacy=0x7f32a80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:23.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f32af38c640 1 -- 172.21.15.22:0/885769809 shutdown_connections 2024-03-29T11:02:23.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.827+0000 7f32af38c640 1 -- 172.21.15.22:0/885769809 >> 172.21.15.22:0/885769809 conn(0x7f32a8066a00 msgr2=0x7f32a8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:23.810 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.828+0000 7f32af38c640 1 -- 172.21.15.22:0/885769809 shutdown_connections 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.828+0000 7f32af38c640 1 -- 172.21.15.22:0/885769809 wait complete. 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.829+0000 7f32af38c640 10 monclient: build_initial_monmap 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.829+0000 7f32af38c640 10 monclient: monmap: 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:23.811 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.829+0000 7f32af38c640 1 librados: starting msgr at 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.829+0000 7f32af38c640 1 librados: starting objecter 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.830+0000 7f32af38c640 1 Processor -- start 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.830+0000 7f32af38c640 1 -- start start 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.830+0000 7f32af38c640 1 librados: setting wanted keys 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.830+0000 7f32af38c640 1 librados: calling monclient init 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.830+0000 7f32af38c640 10 monclient: init 2024-03-29T11:02:23.812 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.831+0000 7f32af38c640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:23.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.831+0000 7f32af38c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:23.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.831+0000 7f32af38c640 10 monclient(hunting): picked mon.noname-a con 0x7f32a80b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:23.813 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.831+0000 7f32af38c640 10 monclient(hunting): picked mon.noname-b con 0x7f32a81422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.834+0000 7f32ad128640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f32a80b17d0 0x7f32a81daba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:34837/0 (socket says 172.21.15.22:34837) 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.834+0000 7f32ad128640 1 -- 172.21.15.22:0/256903781 learned_addr learned my addr 172.21.15.22:0/256903781 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.834+0000 7f329dffb640 10 client.?.objecter ms_handle_connect 0x7f32a80b17d0 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.834+0000 7f329dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.835+0000 7f32af38c640 10 monclient(hunting): picked mon.noname-c con 0x7f32a8142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.835+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80abc90 con 0x7f32a80b17d0 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.835+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80aba50 con 0x7f32a81422b0 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.835+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32a80aca80 con 0x7f32a8142690 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.835+0000 7f32af38c640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:23.816 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.835+0000 7f32af38c640 10 monclient(hunting): authenticate will time out at 1245.140838s 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548070826 0 0) 0x7f32a80abc90 con 0x7f32a80b17d0 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f328c004eb0 con 0x7f32a80b17d0 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 client.?.objecter ms_handle_connect 0x7f32a8142690 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1381004114 0 0) 0x7f328c004eb0 con 0x7f32a80b17d0 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32a80abc90 con 0x7f32a80b17d0 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f32a4001960 con 0x7f32a80b17d0 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:23.821 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): dump: 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:23.822 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.836+0000 7f329dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3526741240 0 0) 0x7f32a80abc90 con 0x7f32a80b17d0 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 >> v1:172.21.15.22:6790/0 conn(0x7f32a8142690 legacy=0x7f32a81e1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 >> v1:172.21.15.26:6789/0 conn(0x7f32a81422b0 legacy=0x7f32a81db2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 1 monclient: found mon.a 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32a80acd60 con 0x7f32a80b17d0 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:23.823 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f32a4001c40 con 0x7f32a80b17d0 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f32a4003a40 con 0x7f32a80b17d0 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.837+0000 7f329dffb640 10 monclient: dump: 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:23.824 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.838+0000 7f32af38c640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.838+0000 7f32af38c640 10 monclient: _renew_subs 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.838+0000 7f32af38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.838+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32a80ad040 con 0x7f32a80b17d0 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.838+0000 7f32af38c640 10 client.4482.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.838+0000 7f32af38c640 10 monclient: _renew_subs 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.839+0000 7f32af38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.839+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f32a8132500 con 0x7f32a80b17d0 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.839+0000 7f32af38c640 1 librados: init done 2024-03-29T11:02:23.825 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.839+0000 7f329effd640 4 set_mon_vals no callback set 2024-03-29T11:02:23.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.842+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f32a4003ba0 con 0x7f32a80b17d0 2024-03-29T11:02:23.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f32a4084b10 con 0x7f32a80b17d0 2024-03-29T11:02:23.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f32ac927640 10 monclient: get_auth_request con 0x7f328c06dc20 auth_method 0 2024-03-29T11:02:23.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f329dffb640 10 client.4482.objecter ms_dispatch 0x7f32a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:23.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f329dffb640 3 client.4482.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:23.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f329dffb640 3 client.4482.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:23.827 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f32af38c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:23.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f32af38c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:23.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f32af38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:23.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3288005080 con 0x7f32a80b17d0 2024-03-29T11:02:23.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f329dffb640 20 client.4482.objecter dump_active .. 0 homeless 2024-03-29T11:02:23.828 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.843+0000 7f329dffb640 10 client.4482.objecter ms_handle_connect 0x7f328c06dc20 2024-03-29T11:02:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.851+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32a40503f0 con 0x7f32a80b17d0 2024-03-29T11:02:23.831 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.851+0000 7f329dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:23.832 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:23.851+0000 7f329dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:24.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.194+0000 7f32af38c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:24.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.194+0000 7f32af38c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:24.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.194+0000 7f32af38c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:24.175 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.194+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3288005330 con 0x7f32a80b17d0 2024-03-29T11:02:24.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.195+0000 7f329dffb640 1 -- 172.21.15.22:0/256903781 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f32a4052320 con 0x7f32a80b17d0 2024-03-29T11:02:24.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.195+0000 7f329dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:24.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.195+0000 7f329dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:24.176 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:24.176 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:24.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 10 librados: watch_flush enter 2024-03-29T11:02:24.179 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 10 librados: watch_flush exit 2024-03-29T11:02:24.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 20 client.4482.objecter shutdown clearing up homeless session... 2024-03-29T11:02:24.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 10 client.4482.objecter successfully canceled tick 2024-03-29T11:02:24.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 >> v1:172.21.15.22:6816/380221644 conn(0x7f328c06dc20 legacy=0x7f328c0700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:24.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 10 monclient: shutdown 2024-03-29T11:02:24.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:24.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.199+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 >> v1:172.21.15.22:6789/0 conn(0x7f32a80b17d0 legacy=0x7f32a81daba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:24.180 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.200+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 shutdown_connections 2024-03-29T11:02:24.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.200+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 >> 172.21.15.22:0/256903781 conn(0x7f32a8066a00 msgr2=0x7f32a80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:24.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.200+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 shutdown_connections 2024-03-29T11:02:24.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.201+0000 7f32af38c640 1 -- 172.21.15.22:0/256903781 wait complete. 2024-03-29T11:02:24.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:24.203+0000 7f32af38c640 1 librados: shutdown 2024-03-29T11:02:24.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:31.012 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.179+0000 7fb7c3a8d640 10 monclient: get_monmap_and_config 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.179+0000 7fb7c3a8d640 10 monclient: build_initial_monmap 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.179+0000 7fb7c3a8d640 10 monclient: monmap: 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:31.181220+0000 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:31.181220+0000 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:31.160 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:31.161 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:31.161 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:31.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.180+0000 7fb7c3a8d640 1 Processor -- start 2024-03-29T11:02:31.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.181+0000 7fb7c3a8d640 1 -- start start 2024-03-29T11:02:31.161 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.181+0000 7fb7c3a8d640 10 monclient: init 2024-03-29T11:02:31.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:31.162 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7bc0b0500 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7bc0b08e0 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7bc0aec70 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc0ae7e0 con 0x7fb7bc0b0500 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc13daf0 con 0x7fb7bc0b08e0 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc0accf0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c3a8d640 10 monclient(hunting): authenticate will time out at 1252.487342s 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c202a640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb7bc0aec70 0x7fb7bc13ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:45674/0 (socket says 172.21.15.22:45674) 2024-03-29T11:02:31.164 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.182+0000 7fb7c202a640 1 -- 172.21.15.22:0/62704399 learned_addr learned my addr 172.21.15.22:0/62704399 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.185+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535529296 0 0) 0x7fb7bc0accf0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.185+0000 7fb7c0827640 10 monclient(hunting): _negotiate 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.185+0000 7fb7c0827640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.185+0000 7fb7c0827640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.185+0000 7fb7c0827640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.185+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a8003520 con 0x7fb7bc0aec70 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192307518 0 0) 0x7fb7a8003520 con 0x7fb7bc0aec70 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 >> v1:172.21.15.26:6789/0 conn(0x7fb7bc0b08e0 legacy=0x7fb7bc13d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 >> v1:172.21.15.22:6789/0 conn(0x7fb7bc0b0500 legacy=0x7fb7bc072050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: _finish_hunting 0 2024-03-29T11:02:31.167 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 monclient: found mon.noname-b 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc140620 con 0x7fb7bc0aec70 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: _finish_auth 0 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: _check_auth_tickets 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb7b8001690 con 0x7fb7bc0aec70 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: dump: 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.168 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7b80018b0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 1 -- 172.21.15.22:0/62704399 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb7b8001c40 con 0x7fb7bc0aec70 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c0827640 10 monclient: dump: 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:31.169 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c3a8d640 5 monclient: authenticate success, global_id 4619 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c3a8d640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:31.170 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.186+0000 7fb7c3a8d640 4 set_mon_vals no callback set 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.187+0000 7fb7c3a8d640 10 monclient: shutdown 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.187+0000 7fb7c3a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.187+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/62704399 >> v1:172.21.15.22:6790/0 conn(0x7fb7bc0aec70 legacy=0x7fb7bc13ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.187+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/62704399 shutdown_connections 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.187+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/62704399 >> 172.21.15.22:0/62704399 conn(0x7fb7bc06b660 msgr2=0x7fb7bc06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.189+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/62704399 shutdown_connections 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.189+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/62704399 wait complete. 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.190+0000 7fb7c3a8d640 10 monclient: build_initial_monmap 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.190+0000 7fb7c3a8d640 10 monclient: monmap: 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:31.172 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.191+0000 7fb7c3a8d640 1 librados: starting msgr at 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.191+0000 7fb7c3a8d640 1 librados: starting objecter 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.191+0000 7fb7c3a8d640 1 Processor -- start 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.191+0000 7fb7c3a8d640 1 -- start start 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.191+0000 7fb7c3a8d640 1 librados: setting wanted keys 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.191+0000 7fb7c3a8d640 1 librados: calling monclient init 2024-03-29T11:02:31.173 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.191+0000 7fb7c3a8d640 10 monclient: init 2024-03-29T11:02:31.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.192+0000 7fb7c3a8d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:31.174 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.192+0000 7fb7c3a8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:02:31.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.196+0000 7fb7c3a8d640 10 monclient(hunting): picked mon.noname-b con 0x7fb7bc0aec70 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:31.176 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.196+0000 7fb7c3a8d640 10 monclient(hunting): picked mon.noname-c con 0x7fb7bc0b0500 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:31.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.197+0000 7fb7c1028640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fb7bc0b0500 0x7fb7bc11b1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:46070/0 (socket says 172.21.15.22:46070) 2024-03-29T11:02:31.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.199+0000 7fb7c1829640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fb7bc0aec70 0x7fb7bc1d3b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:56655/0 (socket says 172.21.15.22:56655) 2024-03-29T11:02:31.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.199+0000 7fb7c3a8d640 10 monclient(hunting): picked mon.noname-a con 0x7fb7bc13a810 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:31.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7c3a8d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc1d3680 con 0x7fb7bc13a810 2024-03-29T11:02:31.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc11ee10 con 0x7fb7bc0aec70 2024-03-29T11:02:31.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7bc1d31a0 con 0x7fb7bc0b0500 2024-03-29T11:02:31.181 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7c3a8d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7c3a8d640 10 monclient(hunting): authenticate will time out at 1252.505205s 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7c1028640 1 -- 172.21.15.22:0/66499857 learned_addr learned my addr 172.21.15.22:0/66499857 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 client.?.objecter ms_handle_connect 0x7fb7bc0aec70 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 client.?.objecter ms_handle_connect 0x7fb7bc13a810 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 client.?.objecter ms_handle_connect 0x7fb7bc0b0500 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319171703 0 0) 0x7fb7bc11ee10 con 0x7fb7bc0aec70 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:31.182 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:02:31.183 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7a0004eb0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.200+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3506875067 0 0) 0x7fb7a0004eb0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb7bc11ee10 con 0x7fb7bc0aec70 2024-03-29T11:02:31.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb7ac001970 con 0x7fb7bc0aec70 2024-03-29T11:02:31.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:31.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:31.185 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient(hunting): dump: 2024-03-29T11:02:31.185 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2314979853 0 0) 0x7fb7bc11ee10 con 0x7fb7bc0aec70 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 >> v1:172.21.15.22:6790/0 conn(0x7fb7bc0b0500 legacy=0x7fb7bc11b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 >> v1:172.21.15.22:6789/0 conn(0x7fb7bc13a810 legacy=0x7fb7bc139b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:31.186 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 1 monclient: found mon.b 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7bc1d3330 con 0x7fb7bc0aec70 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient: _finish_auth 0 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.201+0000 7fb7b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 10 monclient: _renew_subs 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7bc0702b0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 10 monclient: _renew_subs 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:31.187 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7bc13a460 con 0x7fb7bc0aec70 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7c3a8d640 1 librados: init done 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7ac001ad0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7b3fff640 4 set_mon_vals no callback set 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fb7ac003a60 con 0x7fb7bc0aec70 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.204+0000 7fb7b27fc640 10 monclient: dump: 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:31.188 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:31.189 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:31.189 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:31.189 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.205+0000 7fb7c3a8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.205+0000 7fb7c3a8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.205+0000 7fb7c3a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.205+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7a4005080 con 0x7fb7bc0aec70 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.208+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fb7ac003bc0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.209+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fb7ac084ae0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.209+0000 7fb7b27fc640 10 client.4459.objecter ms_dispatch 0x7fb7bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.209+0000 7fb7b27fc640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.209+0000 7fb7b27fc640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.209+0000 7fb7b27fc640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:02:31.190 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.209+0000 7fb7c1028640 10 monclient: get_auth_request con 0x7fb7a006dc40 auth_method 0 2024-03-29T11:02:31.191 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.210+0000 7fb7b27fc640 10 client.4459.objecter ms_handle_connect 0x7fb7a006dc40 2024-03-29T11:02:31.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.217+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb7ac0501f0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.217+0000 7fb7b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:31.197 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.217+0000 7fb7b27fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:31.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.546+0000 7fb7c3a8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:31.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.546+0000 7fb7c3a8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:31.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.546+0000 7fb7c3a8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:02:31.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.546+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7a4005330 con 0x7fb7bc0aec70 2024-03-29T11:02:31.530 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:31.530 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:31.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.548+0000 7fb7b27fc640 1 -- 172.21.15.22:0/66499857 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fb7ac0522f0 con 0x7fb7bc0aec70 2024-03-29T11:02:31.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.548+0000 7fb7b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:31.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.548+0000 7fb7b27fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:31.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 10 librados: watch_flush enter 2024-03-29T11:02:31.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 10 librados: watch_flush exit 2024-03-29T11:02:31.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:02:31.532 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:02:31.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 >> v1:172.21.15.22:6816/380221644 conn(0x7fb7a006dc40 legacy=0x7fb7a0070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:31.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 10 monclient: shutdown 2024-03-29T11:02:31.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:31.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 >> v1:172.21.15.26:6789/0 conn(0x7fb7bc0aec70 legacy=0x7fb7bc1d3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:31.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 shutdown_connections 2024-03-29T11:02:31.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.551+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 >> 172.21.15.22:0/66499857 conn(0x7fb7bc06b660 msgr2=0x7fb7bc071e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:31.533 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.552+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 shutdown_connections 2024-03-29T11:02:31.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.554+0000 7fb7c3a8d640 1 -- 172.21.15.22:0/66499857 wait complete. 2024-03-29T11:02:31.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:31.554+0000 7fb7c3a8d640 1 librados: shutdown 2024-03-29T11:02:31.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:38.358 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:38.512 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.529+0000 7f6bf8cf2640 10 monclient: get_monmap_and_config 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.529+0000 7f6bf8cf2640 10 monclient: build_initial_monmap 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.529+0000 7f6bf8cf2640 10 monclient: monmap: 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:38.529943+0000 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:38.529943+0000 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:38.513 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.531+0000 7f6bf8cf2640 1 Processor -- start 2024-03-29T11:02:38.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.531+0000 7f6bf8cf2640 1 -- start start 2024-03-29T11:02:38.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.531+0000 7f6bf8cf2640 10 monclient: init 2024-03-29T11:02:38.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:38.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:38.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf40aac70 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:38.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf40ab050 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:38.515 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf40ad160 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf40abb60 con 0x7f6bf40ad160 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf40ac070 con 0x7f6bf40ab050 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf40ac350 con 0x7f6bf40aac70 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.534+0000 7f6bf8cf2640 10 monclient(hunting): authenticate will time out at 1259.838407s 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf37fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6bf40aac70 0x7f6bf40aca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52050/0 (socket says 172.21.15.22:52050) 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf37fe640 1 -- 172.21.15.22:0/2394251578 learned_addr learned my addr 172.21.15.22:0/2394251578 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900760592 0 0) 0x7f6bf40ac350 con 0x7f6bf40aac70 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:02:38.516 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf27fc640 10 monclient(hunting): my global_id is 4625 2024-03-29T11:02:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.536+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6bdc003520 con 0x7f6bf40aac70 2024-03-29T11:02:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.537+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1516393999 0 0) 0x7f6bdc003520 con 0x7f6bf40aac70 2024-03-29T11:02:38.517 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.537+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 >> v1:172.21.15.26:6789/0 conn(0x7f6bf40ab050 legacy=0x7f6bf40ae460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 >> v1:172.21.15.22:6789/0 conn(0x7f6bf40ad160 legacy=0x7f6bf415bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 1 monclient: found mon.noname-b 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf4155b00 con 0x7f6bf40aac70 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 10 monclient: _finish_auth 0 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6be8001690 con 0x7f6bf40aac70 2024-03-29T11:02:38.524 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf27fc640 10 monclient: dump: 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf8cf2640 5 monclient: authenticate success, global_id 4625 2024-03-29T11:02:38.525 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.543+0000 7f6bf8cf2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:38.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.544+0000 7f6bf27fc640 1 -- 172.21.15.22:0/2394251578 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6be8001860 con 0x7f6bf40aac70 2024-03-29T11:02:38.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.544+0000 7f6bf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:38.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.544+0000 7f6bf8cf2640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:38.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.544+0000 7f6bf8cf2640 4 set_mon_vals no callback set 2024-03-29T11:02:38.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.545+0000 7f6bf8cf2640 10 monclient: shutdown 2024-03-29T11:02:38.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.545+0000 7f6bf8cf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:38.526 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.545+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/2394251578 >> v1:172.21.15.22:6790/0 conn(0x7f6bf40aac70 legacy=0x7f6bf40aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:38.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.547+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/2394251578 shutdown_connections 2024-03-29T11:02:38.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.547+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/2394251578 >> 172.21.15.22:0/2394251578 conn(0x7f6bf40b3a30 msgr2=0x7f6bf40b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:38.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.547+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/2394251578 shutdown_connections 2024-03-29T11:02:38.527 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.547+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/2394251578 wait complete. 2024-03-29T11:02:38.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.549+0000 7f6bf8cf2640 10 monclient: build_initial_monmap 2024-03-29T11:02:38.529 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.549+0000 7f6bf8cf2640 10 monclient: monmap: 2024-03-29T11:02:38.529 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:38.529 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:38.529 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:38.529 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.550+0000 7f6bf8cf2640 1 librados: starting msgr at 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.550+0000 7f6bf8cf2640 1 librados: starting objecter 2024-03-29T11:02:38.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.550+0000 7f6bf8cf2640 1 Processor -- start 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.550+0000 7f6bf8cf2640 1 -- start start 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.550+0000 7f6bf8cf2640 1 librados: setting wanted keys 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.550+0000 7f6bf8cf2640 1 librados: calling monclient init 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.550+0000 7f6bf8cf2640 10 monclient: init 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.551+0000 7f6bf8cf2640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.551+0000 7f6bf8cf2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.551+0000 7f6bf8cf2640 10 monclient(hunting): picked mon.noname-c con 0x7f6bf40aac70 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.551+0000 7f6bf8cf2640 10 monclient(hunting): picked mon.noname-b con 0x7f6bf40ad160 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:38.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.551+0000 7f6bf37fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6bf40aac70 0x7f6bf406dde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52468/0 (socket says 172.21.15.22:52468) 2024-03-29T11:02:38.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.554+0000 7f6bf37fe640 1 -- 172.21.15.22:0/1597385878 learned_addr learned my addr 172.21.15.22:0/1597385878 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:38.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.554+0000 7f6bf8cf2640 10 monclient(hunting): picked mon.noname-a con 0x7f6bf41477a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:38.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.554+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf41558d0 con 0x7f6bf41477a0 2024-03-29T11:02:38.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.554+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf41484b0 con 0x7f6bf40ad160 2024-03-29T11:02:38.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.554+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bf406d6e0 con 0x7f6bf40aac70 2024-03-29T11:02:38.534 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.554+0000 7f6bf8cf2640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:38.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.554+0000 7f6bf8cf2640 10 monclient(hunting): authenticate will time out at 1259.859056s 2024-03-29T11:02:38.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.555+0000 7f6bdbfff640 10 client.?.objecter ms_handle_connect 0x7f6bf40aac70 2024-03-29T11:02:38.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.555+0000 7f6bdbfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:38.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.557+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520705563 0 0) 0x7f6bf406d6e0 con 0x7f6bf40aac70 2024-03-29T11:02:38.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.557+0000 7f6bdbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:38.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.557+0000 7f6bdbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:38.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.557+0000 7f6bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:38.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.557+0000 7f6bdbfff640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:02:38.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.557+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6be0004eb0 con 0x7f6bf40aac70 2024-03-29T11:02:38.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 139354218 0 0) 0x7f6be0004eb0 con 0x7f6bf40aac70 2024-03-29T11:02:38.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6bf406d6e0 con 0x7f6bf40aac70 2024-03-29T11:02:38.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6be8004ae0 con 0x7f6bf40aac70 2024-03-29T11:02:38.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:38.538 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 10 monclient(hunting): dump: 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3053302024 0 0) 0x7f6bf406d6e0 con 0x7f6bf40aac70 2024-03-29T11:02:38.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 >> v1:172.21.15.26:6789/0 conn(0x7f6bf40ad160 legacy=0x7f6bf4154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:38.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 >> v1:172.21.15.22:6789/0 conn(0x7f6bf41477a0 legacy=0x7f6bf41551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:38.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:38.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 1 monclient: found mon.c 2024-03-29T11:02:38.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.558+0000 7f6bdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:38.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.559+0000 7f6bdbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:38.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.559+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf406d9c0 con 0x7f6bf40aac70 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bdbfff640 10 monclient: _finish_auth 0 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bdbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 5 monclient: authenticate success, global_id 4628 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 10 monclient: _renew_subs 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bf41466d0 con 0x7f6bf40aac70 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 10 client.4628.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 10 monclient: _renew_subs 2024-03-29T11:02:38.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bf4082020 con 0x7f6bf40aac70 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.560+0000 7f6bf8cf2640 1 librados: init done 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.561+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6be8001ad0 con 0x7f6bf40aac70 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.561+0000 7f6bdbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.561+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6be80050d0 con 0x7f6bf40aac70 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.561+0000 7f6bdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.561+0000 7f6bdbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.561+0000 7f6bdbfff640 10 monclient: dump: 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:38.543 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:38.544 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:38.544 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:38.544 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:38.544 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:38.544 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.562+0000 7f6bf1ffb640 4 set_mon_vals no callback set 2024-03-29T11:02:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.563+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f6be8005230 con 0x7f6bf40aac70 2024-03-29T11:02:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bf8cf2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bf8cf2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bf8cf2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:38.545 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd4005080 con 0x7f6bf40aac70 2024-03-29T11:02:38.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bf2ffd640 10 monclient: get_auth_request con 0x7f6be006dc20 auth_method 0 2024-03-29T11:02:38.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f6be8084f10 con 0x7f6bf40aac70 2024-03-29T11:02:38.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bdbfff640 10 client.4628.objecter ms_dispatch 0x7f6bf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:38.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bdbfff640 3 client.4628.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:38.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.564+0000 7f6bdbfff640 3 client.4628.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:38.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.565+0000 7f6bdbfff640 20 client.4628.objecter dump_active .. 0 homeless 2024-03-29T11:02:38.546 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.565+0000 7f6bdbfff640 10 client.4628.objecter ms_handle_connect 0x7f6be006dc20 2024-03-29T11:02:38.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.574+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6be80507f0 con 0x7f6bf40aac70 2024-03-29T11:02:38.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.574+0000 7f6bdbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:38.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.574+0000 7f6bdbfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:38.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.913+0000 7f6bf8cf2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:38.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.913+0000 7f6bf8cf2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:38.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.913+0000 7f6bf8cf2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:38.893 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.913+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bd4005330 con 0x7f6bf40aac70 2024-03-29T11:02:38.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.914+0000 7f6bdbfff640 1 -- 172.21.15.22:0/1597385878 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f6be8052720 con 0x7f6bf40aac70 2024-03-29T11:02:38.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.915+0000 7f6bdbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:38.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.915+0000 7f6bdbfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:38.894 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:38.894 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:38.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 10 librados: watch_flush enter 2024-03-29T11:02:38.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 10 librados: watch_flush exit 2024-03-29T11:02:38.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 20 client.4628.objecter shutdown clearing up homeless session... 2024-03-29T11:02:38.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 10 client.4628.objecter successfully canceled tick 2024-03-29T11:02:38.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 >> v1:172.21.15.22:6816/380221644 conn(0x7f6be006dc20 legacy=0x7f6be00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:38.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 10 monclient: shutdown 2024-03-29T11:02:38.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:38.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 >> v1:172.21.15.22:6790/0 conn(0x7f6bf40aac70 legacy=0x7f6bf406dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:38.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 shutdown_connections 2024-03-29T11:02:38.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.918+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 >> 172.21.15.22:0/1597385878 conn(0x7f6bf40b3a30 msgr2=0x7f6bf4069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:38.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.919+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 shutdown_connections 2024-03-29T11:02:38.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.919+0000 7f6bf8cf2640 1 -- 172.21.15.22:0/1597385878 wait complete. 2024-03-29T11:02:38.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:38.919+0000 7f6bf8cf2640 1 librados: shutdown 2024-03-29T11:02:38.915 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:45.722 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:45.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.896+0000 7f0b9194b640 10 monclient: get_monmap_and_config 2024-03-29T11:02:45.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.896+0000 7f0b9194b640 10 monclient: build_initial_monmap 2024-03-29T11:02:45.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.896+0000 7f0b9194b640 10 monclient: monmap: 2024-03-29T11:02:45.877 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:45.877 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:45.877 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:45.897164+0000 2024-03-29T11:02:45.877 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:45.897164+0000 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.897+0000 7f0b9194b640 1 Processor -- start 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.897+0000 7f0b9194b640 1 -- start start 2024-03-29T11:02:45.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.897+0000 7f0b9194b640 10 monclient: init 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8c0b4100 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b8c13ee90 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8c0af900 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c0afd10 con 0x7f0b8c13ee90 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c0af6f0 con 0x7f0b8c0b4100 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c0aaf60 con 0x7f0b8c0af900 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:45.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b9194b640 10 monclient(hunting): authenticate will time out at 1267.202818s 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b8b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0b8c0af900 0x7f0b8c13b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52716/0 (socket says 172.21.15.22:52716) 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.898+0000 7f0b8b7fe640 1 -- 172.21.15.22:0/3425196585 learned_addr learned my addr 172.21.15.22:0/3425196585 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.901+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107606870 0 0) 0x7f0b8c0aaf60 con 0x7f0b8c0af900 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.901+0000 7f0b89ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.901+0000 7f0b89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.901+0000 7f0b89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.901+0000 7f0b89ffb640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:02:45.881 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.901+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b74003520 con 0x7f0b8c0af900 2024-03-29T11:02:45.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.902+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784630672 0 0) 0x7f0b74003520 con 0x7f0b8c0af900 2024-03-29T11:02:45.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.902+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 >> v1:172.21.15.26:6789/0 conn(0x7f0b8c0b4100 legacy=0x7f0b8c0a9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:45.882 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.902+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 >> v1:172.21.15.22:6789/0 conn(0x7f0b8c13ee90 legacy=0x7f0b8c0adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 1 monclient: found mon.noname-b 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c13e970 con 0x7f0b8c0af900 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 10 monclient: _finish_auth 0 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0b84001a10 con 0x7f0b8c0af900 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:45.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.904+0000 7f0b89ffb640 10 monclient: dump: 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b84001be0 con 0x7f0b8c0af900 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:45.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b89ffb640 1 -- 172.21.15.22:0/3425196585 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0b84002030 con 0x7f0b8c0af900 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b89ffb640 10 monclient: dump: 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b9194b640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:02:45.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b9194b640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:45.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.905+0000 7f0b9194b640 4 set_mon_vals no callback set 2024-03-29T11:02:45.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.907+0000 7f0b9194b640 10 monclient: shutdown 2024-03-29T11:02:45.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.907+0000 7f0b9194b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:45.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.907+0000 7f0b9194b640 1 -- 172.21.15.22:0/3425196585 >> v1:172.21.15.22:6790/0 conn(0x7f0b8c0af900 legacy=0x7f0b8c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:45.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.907+0000 7f0b9194b640 1 -- 172.21.15.22:0/3425196585 shutdown_connections 2024-03-29T11:02:45.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.907+0000 7f0b9194b640 1 -- 172.21.15.22:0/3425196585 >> 172.21.15.22:0/3425196585 conn(0x7f0b8c0b35e0 msgr2=0x7f0b8c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:45.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.907+0000 7f0b9194b640 1 -- 172.21.15.22:0/3425196585 shutdown_connections 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.907+0000 7f0b9194b640 1 -- 172.21.15.22:0/3425196585 wait complete. 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.908+0000 7f0b9194b640 10 monclient: build_initial_monmap 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.908+0000 7f0b9194b640 10 monclient: monmap: 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:45.890 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.908+0000 7f0b9194b640 1 librados: starting msgr at 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.908+0000 7f0b9194b640 1 librados: starting objecter 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.908+0000 7f0b9194b640 1 Processor -- start 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 1 -- start start 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 1 librados: setting wanted keys 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 1 librados: calling monclient init 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient: init 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:45.891 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient(hunting): picked mon.noname-a con 0x7f0b8c0af900 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient(hunting): picked mon.noname-b con 0x7f0b8c0b4100 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient(hunting): picked mon.noname-c con 0x7f0b8c13ee90 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c070910 con 0x7f0b8c0af900 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c070bf0 con 0x7f0b8c0b4100 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b8c06f0a0 con 0x7f0b8c13ee90 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.909+0000 7f0b9194b640 10 monclient(hunting): authenticate will time out at 1267.214058s 2024-03-29T11:02:45.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.910+0000 7f0b8b7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f0b8c13ee90 0x7f0b8c1e26e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52932/0 (socket says 172.21.15.22:52932) 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.910+0000 7f0b8b7fe640 1 -- 172.21.15.22:0/4130548542 learned_addr learned my addr 172.21.15.22:0/4130548542 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.910+0000 7f0b73fff640 10 client.?.objecter ms_handle_connect 0x7f0b8c13ee90 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.910+0000 7f0b73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061608234 0 0) 0x7f0b8c06f0a0 con 0x7f0b8c13ee90 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b78004eb0 con 0x7f0b8c13ee90 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 client.?.objecter ms_handle_connect 0x7f0b8c0af900 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771362841 0 0) 0x7f0b8c070910 con 0x7f0b8c0af900 2024-03-29T11:02:45.898 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): _negotiate 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.913+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b780051d0 con 0x7f0b8c0af900 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1759777032 0 0) 0x7f0b780051d0 con 0x7f0b8c0af900 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0b78005640 con 0x7f0b8c0af900 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0b7c0018b0 con 0x7f0b8c0af900 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 10 monclient(hunting): dump: 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:45.899 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 567002181 0 0) 0x7f0b78005640 con 0x7f0b8c0af900 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 >> v1:172.21.15.22:6790/0 conn(0x7f0b8c13ee90 legacy=0x7f0b8c1e26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.914+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 >> v1:172.21.15.26:6789/0 conn(0x7f0b8c0b4100 legacy=0x7f0b8c1defb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 10 monclient: _finish_hunting 0 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 1 monclient: found mon.a 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b8c142610 con 0x7f0b8c0af900 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 10 monclient: _finish_auth 0 2024-03-29T11:02:45.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 10 monclient: _check_auth_tickets 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 5 monclient: authenticate success, global_id 4491 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 10 monclient: _renew_subs 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b8c071720 con 0x7f0b8c0af900 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 10 monclient: _renew_subs 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b8c1318f0 con 0x7f0b8c0af900 2024-03-29T11:02:45.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.916+0000 7f0b9194b640 1 librados: init done 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b9194b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b9194b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b9194b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b8c06ef00 con 0x7f0b8c0af900 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0b7c0035f0 con 0x7f0b8c0af900 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b73fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0b7c0018b0 con 0x7f0b8c0af900 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b73fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b73fff640 10 monclient: dump: 2024-03-29T11:02:45.907 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.918+0000 7f0b88ff9640 4 set_mon_vals no callback set 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.920+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f0b7c0195d0 con 0x7f0b8c0af900 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.921+0000 7f0b8a7fc640 10 monclient: get_auth_request con 0x7f0b7806e5a0 auth_method 0 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.921+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0b7c052f70 con 0x7f0b8c0af900 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.921+0000 7f0b73fff640 10 client.4491.objecter ms_dispatch 0x7f0b8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:45.908 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.921+0000 7f0b73fff640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:45.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.921+0000 7f0b73fff640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:45.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.921+0000 7f0b73fff640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-29T11:02:45.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.923+0000 7f0b73fff640 10 client.4491.objecter ms_handle_connect 0x7f0b7806e5a0 2024-03-29T11:02:45.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.926+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0b7c04f210 con 0x7f0b8c0af900 2024-03-29T11:02:45.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.926+0000 7f0b73fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:45.909 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:45.926+0000 7f0b73fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:46.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.252+0000 7f0b9194b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:46.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.252+0000 7f0b9194b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:46.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.252+0000 7f0b9194b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:02:46.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.252+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b8c06f2e0 con 0x7f0b8c0af900 2024-03-29T11:02:46.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.253+0000 7f0b73fff640 1 -- 172.21.15.22:0/4130548542 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0b7c04f210 con 0x7f0b8c0af900 2024-03-29T11:02:46.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.253+0000 7f0b73fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:46.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.253+0000 7f0b73fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:46.233 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:46.233 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.256+0000 7f0b9194b640 10 librados: watch_flush enter 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 10 librados: watch_flush exit 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 10 client.4491.objecter successfully canceled tick 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 >> v1:172.21.15.22:6816/380221644 conn(0x7f0b7806e5a0 legacy=0x7f0b78070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 10 monclient: shutdown 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:46.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 >> v1:172.21.15.22:6789/0 conn(0x7f0b8c0af900 legacy=0x7f0b8c071010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:46.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 shutdown_connections 2024-03-29T11:02:46.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 >> 172.21.15.22:0/4130548542 conn(0x7f0b8c0b35e0 msgr2=0x7f0b8c0b11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:46.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 shutdown_connections 2024-03-29T11:02:46.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 1 -- 172.21.15.22:0/4130548542 wait complete. 2024-03-29T11:02:46.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:46.257+0000 7f0b9194b640 1 librados: shutdown 2024-03-29T11:02:46.253 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:02:53.061 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.250+0000 7fccef04d640 10 monclient: get_monmap_and_config 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.250+0000 7fccef04d640 10 monclient: build_initial_monmap 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.250+0000 7fccef04d640 10 monclient: monmap: 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:02:53.251357+0000 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:02:53.251357+0000 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:53.230 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:53.231 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:02:53.231 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:02:53.231 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:53.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.251+0000 7fccef04d640 1 Processor -- start 2024-03-29T11:02:53.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.251+0000 7fccef04d640 1 -- start start 2024-03-29T11:02:53.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.251+0000 7fccef04d640 10 monclient: init 2024-03-29T11:02:53.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.252+0000 7fccef04d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:53.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.252+0000 7fccef04d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:02:53.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.252+0000 7fccef04d640 10 monclient(hunting): picked mon.noname-b con 0x7fcce80b1f60 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:53.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.252+0000 7fccef04d640 10 monclient(hunting): picked mon.noname-c con 0x7fcce8071eb0 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:53.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccef04d640 10 monclient(hunting): picked mon.noname-a con 0x7fcce8143d10 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:53.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccef04d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce806ef80 con 0x7fcce8143d10 2024-03-29T11:02:53.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccef04d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce806f260 con 0x7fcce8071eb0 2024-03-29T11:02:53.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccef04d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce80b40c0 con 0x7fcce80b1f60 2024-03-29T11:02:53.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccef04d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:53.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccef04d640 10 monclient(hunting): authenticate will time out at 1274.559495s 2024-03-29T11:02:53.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccecde9640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fcce80b1f60 0x7fcce8143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35822/0 (socket says 172.21.15.22:35822) 2024-03-29T11:02:53.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.255+0000 7fccecde9640 1 -- 172.21.15.22:0/123503167 learned_addr learned my addr 172.21.15.22:0/123503167 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:53.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2644756835 0 0) 0x7fcce806f260 con 0x7fcce8071eb0 2024-03-29T11:02:53.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:53.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:53.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:53.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:02:53.236 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd4003520 con 0x7fcce8071eb0 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1792706842 0 0) 0x7fcce80b40c0 con 0x7fcce80b1f60 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd40048a0 con 0x7fcce80b1f60 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980534577 0 0) 0x7fcce806ef80 con 0x7fcce8143d10 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:53.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.256+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccd4004be0 con 0x7fcce8143d10 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1583160958 0 0) 0x7fccd40048a0 con 0x7fcce80b1f60 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 >> v1:172.21.15.26:6789/0 conn(0x7fcce8071eb0 legacy=0x7fcce8146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 >> v1:172.21.15.22:6789/0 conn(0x7fcce8143d10 legacy=0x7fcce8146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 monclient: found mon.noname-b 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce8155d70 con 0x7fcce80b1f60 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: _finish_auth 0 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fccd00015f0 con 0x7fcce80b1f60 2024-03-29T11:02:53.238 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: dump: 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fccef04d640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:02:53.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fccef04d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:02:53.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd00017d0 con 0x7fcce80b1f60 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 1 -- 172.21.15.22:0/123503167 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fccd0001c20 con 0x7fcce80b1f60 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.257+0000 7fcce77fe640 10 monclient: dump: 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:53.242 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.261+0000 7fccef04d640 10 monclient: get_monmap_and_config success 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.261+0000 7fccef04d640 4 set_mon_vals no callback set 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.262+0000 7fccef04d640 10 monclient: shutdown 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.262+0000 7fccef04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.262+0000 7fccef04d640 1 -- 172.21.15.22:0/123503167 >> v1:172.21.15.22:6790/0 conn(0x7fcce80b1f60 legacy=0x7fcce8143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.262+0000 7fccef04d640 1 -- 172.21.15.22:0/123503167 shutdown_connections 2024-03-29T11:02:53.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.262+0000 7fccef04d640 1 -- 172.21.15.22:0/123503167 >> 172.21.15.22:0/123503167 conn(0x7fcce8066a00 msgr2=0x7fcce8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:53.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.264+0000 7fccef04d640 1 -- 172.21.15.22:0/123503167 shutdown_connections 2024-03-29T11:02:53.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.264+0000 7fccef04d640 1 -- 172.21.15.22:0/123503167 wait complete. 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.265+0000 7fccef04d640 10 monclient: build_initial_monmap 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.265+0000 7fccef04d640 10 monclient: monmap: 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:02:53.245 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.266+0000 7fccef04d640 1 librados: starting msgr at 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.266+0000 7fccef04d640 1 librados: starting objecter 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.266+0000 7fccef04d640 1 Processor -- start 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.266+0000 7fccef04d640 1 -- start start 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.266+0000 7fccef04d640 1 librados: setting wanted keys 2024-03-29T11:02:53.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.266+0000 7fccef04d640 1 librados: calling monclient init 2024-03-29T11:02:53.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.266+0000 7fccef04d640 10 monclient: init 2024-03-29T11:02:53.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.267+0000 7fccef04d640 10 monclient: _reopen_session rank -1 2024-03-29T11:02:53.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.267+0000 7fccef04d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.267+0000 7fccef04d640 10 monclient(hunting): picked mon.noname-a con 0x7fcce80b1f60 addr v1:172.21.15.22:6789/0 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.267+0000 7fccef04d640 10 monclient(hunting): picked mon.noname-b con 0x7fcce8143d10 addr v1:172.21.15.26:6789/0 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.270+0000 7fccef04d640 10 monclient(hunting): picked mon.noname-c con 0x7fcce80709b0 addr v1:172.21.15.22:6790/0 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.270+0000 7fccef04d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce81495b0 con 0x7fcce80b1f60 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.270+0000 7fccef04d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce8149890 con 0x7fcce8143d10 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.270+0000 7fccef04d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce814aec0 con 0x7fcce80709b0 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.270+0000 7fccef04d640 10 monclient(hunting): _renew_subs 2024-03-29T11:02:53.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.270+0000 7fccef04d640 10 monclient(hunting): authenticate will time out at 1274.574692s 2024-03-29T11:02:53.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.273+0000 7fcced5ea640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fcce80709b0 0x7fcce81e60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:36046/0 (socket says 172.21.15.22:36046) 2024-03-29T11:02:53.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.273+0000 7fcced5ea640 1 -- 172.21.15.22:0/1058052519 learned_addr learned my addr 172.21.15.22:0/1058052519 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:02:53.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 client.?.objecter ms_handle_connect 0x7fcce80709b0 2024-03-29T11:02:53.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:53.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 client.?.objecter ms_handle_connect 0x7fcce80b1f60 2024-03-29T11:02:53.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:53.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662574298 0 0) 0x7fcce814aec0 con 0x7fcce80709b0 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc8004eb0 con 0x7fcce80709b0 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773415330 0 0) 0x7fcce81495b0 con 0x7fcce80b1f60 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:02:53.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.274+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcce814aec0 con 0x7fcce80b1f60 2024-03-29T11:02:53.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 client.?.objecter ms_handle_connect 0x7fcce8143d10 2024-03-29T11:02:53.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:02:53.255 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896219586 0 0) 0x7fcce8149890 con 0x7fcce8143d10 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc8005360 con 0x7fcce8143d10 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1200683735 0 0) 0x7fccc8004eb0 con 0x7fcce80709b0 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcce8149890 con 0x7fcce80709b0 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fccd8001c10 con 0x7fcce80709b0 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): dump: 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:53.256 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3308037182 0 0) 0x7fcce814aec0 con 0x7fcce80b1f60 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc8004eb0 con 0x7fcce80b1f60 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fccd0001c20 con 0x7fcce80b1f60 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:02:53.257 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.275+0000 7fcce57fa640 10 monclient(hunting): dump: 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3522403679 0 0) 0x7fcce8149890 con 0x7fcce80709b0 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 >> v1:172.21.15.26:6789/0 conn(0x7fcce8143d10 legacy=0x7fcce8149d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 >> v1:172.21.15.22:6789/0 conn(0x7fcce80b1f60 legacy=0x7fcce8147790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:53.258 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 1 monclient: found mon.c 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce813b9c0 con 0x7fcce80709b0 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: _finish_auth 0 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 10 monclient: _renew_subs 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce814b130 con 0x7fcce80709b0 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 10 client.4640.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:02:53.259 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 10 monclient: _renew_subs 2024-03-29T11:02:53.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:53.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce8148fb0 con 0x7fcce80709b0 2024-03-29T11:02:53.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fccef04d640 1 librados: init done 2024-03-29T11:02:53.260 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccd80020e0 con 0x7fcce80709b0 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fccd8003b30 con 0x7fcce80709b0 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce6ffd640 4 set_mon_vals no callback set 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.276+0000 7fcce57fa640 10 monclient: dump: 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:02:53.261 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.281+0000 7fccef04d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.281+0000 7fccef04d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.281+0000 7fccef04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.281+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fccd8003c90 con 0x7fcce80709b0 2024-03-29T11:02:53.262 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.281+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcccc005080 con 0x7fcce80709b0 2024-03-29T11:02:53.263 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.282+0000 7fccecde9640 10 monclient: get_auth_request con 0x7fccc806df90 auth_method 0 2024-03-29T11:02:53.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.285+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fccd8084d10 con 0x7fcce80709b0 2024-03-29T11:02:53.264 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.285+0000 7fcce57fa640 10 client.4640.objecter ms_dispatch 0x7fcce8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:02:53.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.285+0000 7fcce57fa640 3 client.4640.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:02:53.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.285+0000 7fcce57fa640 3 client.4640.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:02:53.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.285+0000 7fcce57fa640 20 client.4640.objecter dump_active .. 0 homeless 2024-03-29T11:02:53.265 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.285+0000 7fcce57fa640 10 client.4640.objecter ms_handle_connect 0x7fccc806df90 2024-03-29T11:02:53.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.291+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fccd8050420 con 0x7fcce80709b0 2024-03-29T11:02:53.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.291+0000 7fcce57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:02:53.271 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.291+0000 7fcce57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:02:53.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.650+0000 7fccef04d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:53.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.650+0000 7fccef04d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:53.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.650+0000 7fccef04d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:02:53.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.650+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcccc005330 con 0x7fcce80709b0 2024-03-29T11:02:53.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.651+0000 7fcce57fa640 1 -- 172.21.15.22:0/1058052519 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fccd8052520 con 0x7fcce80709b0 2024-03-29T11:02:53.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.651+0000 7fcce57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:02:53.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.651+0000 7fcce57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:02:53.631 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:02:53.631 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:02:53.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.655+0000 7fccef04d640 10 librados: watch_flush enter 2024-03-29T11:02:53.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.655+0000 7fccef04d640 10 librados: watch_flush exit 2024-03-29T11:02:53.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.655+0000 7fccef04d640 20 client.4640.objecter shutdown clearing up homeless session... 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.655+0000 7fccef04d640 10 client.4640.objecter successfully canceled tick 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 >> v1:172.21.15.22:6816/380221644 conn(0x7fccc806df90 legacy=0x7fccc8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 10 monclient: shutdown 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 >> v1:172.21.15.22:6790/0 conn(0x7fcce80709b0 legacy=0x7fcce81e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 shutdown_connections 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 >> 172.21.15.22:0/1058052519 conn(0x7fcce8066a00 msgr2=0x7fcce80b7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:02:53.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 shutdown_connections 2024-03-29T11:02:53.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.656+0000 7fccef04d640 1 -- 172.21.15.22:0/1058052519 wait complete. 2024-03-29T11:02:53.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:02:53.661+0000 7fccef04d640 1 librados: shutdown 2024-03-29T11:02:53.659 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:00.467 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.641+0000 7f0835d6b640 10 monclient: get_monmap_and_config 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.641+0000 7f0835d6b640 10 monclient: build_initial_monmap 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.641+0000 7f0835d6b640 10 monclient: monmap: 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:00.641879+0000 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:00.641879+0000 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:00.621 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:00.622 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.642+0000 7f0835d6b640 1 Processor -- start 2024-03-29T11:03:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.642+0000 7f0835d6b640 1 -- start start 2024-03-29T11:03:00.622 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.642+0000 7f0835d6b640 10 monclient: init 2024-03-29T11:03:00.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:00.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:00.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f083006d9d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:00.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f083013e820 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:00.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f08300af1d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:00.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08300aed10 con 0x7f083013e820 2024-03-29T11:03:00.624 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08300afd50 con 0x7f083006d9d0 2024-03-29T11:03:00.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08300ab490 con 0x7f08300af1d0 2024-03-29T11:03:00.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:00.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.643+0000 7f0835d6b640 10 monclient(hunting): authenticate will time out at 1281.947870s 2024-03-29T11:03:00.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082ffff640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f08300af1d0 0x7f08300af640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55256/0 (socket says 172.21.15.22:55256) 2024-03-29T11:03:00.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082ffff640 1 -- 172.21.15.22:0/2193197603 learned_addr learned my addr 172.21.15.22:0/2193197603 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:00.625 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036603131 0 0) 0x7f08300ab490 con 0x7f08300af1d0 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082e7fc640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.645+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0818003520 con 0x7f08300af1d0 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.646+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3996085107 0 0) 0x7f0818003520 con 0x7f08300af1d0 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.646+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 >> v1:172.21.15.26:6789/0 conn(0x7f083006d9d0 legacy=0x7f08300b05f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:00.626 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.646+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 >> v1:172.21.15.22:6789/0 conn(0x7f083013e820 legacy=0x7f083013ec50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 1 monclient: found mon.noname-b 2024-03-29T11:03:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:03:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08300ab770 con 0x7f08300af1d0 2024-03-29T11:03:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:00.628 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0828001a10 con 0x7f08300af1d0 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f082e7fc640 10 monclient: dump: 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:00.629 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f0835d6b640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.648+0000 7f0835d6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0828001c70 con 0x7f08300af1d0 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f082e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f082e7fc640 1 -- 172.21.15.22:0/2193197603 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f08280020c0 con 0x7f08300af1d0 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f082e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f082e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f082e7fc640 10 monclient: dump: 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.630 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f0835d6b640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.649+0000 7f0835d6b640 4 set_mon_vals no callback set 2024-03-29T11:03:00.631 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.650+0000 7f0835d6b640 10 monclient: shutdown 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.650+0000 7f0835d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.650+0000 7f0835d6b640 1 -- 172.21.15.22:0/2193197603 >> v1:172.21.15.22:6790/0 conn(0x7f08300af1d0 legacy=0x7f08300af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.650+0000 7f0835d6b640 1 -- 172.21.15.22:0/2193197603 shutdown_connections 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.650+0000 7f0835d6b640 1 -- 172.21.15.22:0/2193197603 >> 172.21.15.22:0/2193197603 conn(0x7f083006b660 msgr2=0x7f083006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.650+0000 7f0835d6b640 1 -- 172.21.15.22:0/2193197603 shutdown_connections 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.650+0000 7f0835d6b640 1 -- 172.21.15.22:0/2193197603 wait complete. 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.651+0000 7f0835d6b640 10 monclient: build_initial_monmap 2024-03-29T11:03:00.632 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.651+0000 7f0835d6b640 10 monclient: monmap: 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.652+0000 7f0835d6b640 1 librados: starting msgr at 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.652+0000 7f0835d6b640 1 librados: starting objecter 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.652+0000 7f0835d6b640 1 Processor -- start 2024-03-29T11:03:00.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.652+0000 7f0835d6b640 1 -- start start 2024-03-29T11:03:00.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.652+0000 7f0835d6b640 1 librados: setting wanted keys 2024-03-29T11:03:00.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.652+0000 7f0835d6b640 1 librados: calling monclient init 2024-03-29T11:03:00.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.652+0000 7f0835d6b640 10 monclient: init 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 10 monclient(hunting): picked mon.noname-c con 0x7f083006d9d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 10 monclient(hunting): picked mon.noname-a con 0x7f08300af1d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 10 monclient(hunting): picked mon.noname-b con 0x7f083013e820 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083006e890 con 0x7f08300af1d0 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083006eb70 con 0x7f083013e820 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.653+0000 7f0835d6b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f083006ee50 con 0x7f083006d9d0 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.654+0000 7f0835d6b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:00.635 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.654+0000 7f0835d6b640 10 monclient(hunting): authenticate will time out at 1281.958101s 2024-03-29T11:03:00.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f082f7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f083006d9d0 0x7f08300b32d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55546/0 (socket says 172.21.15.22:55546) 2024-03-29T11:03:00.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f082f7fe640 1 -- 172.21.15.22:0/4120071167 learned_addr learned my addr 172.21.15.22:0/4120071167 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:00.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 10 client.?.objecter ms_handle_connect 0x7f083006d9d0 2024-03-29T11:03:00.636 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766249108 0 0) 0x7f083006ee50 con 0x7f083006d9d0 2024-03-29T11:03:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:03:00.637 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.657+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f081c004eb0 con 0x7f083006d9d0 2024-03-29T11:03:00.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 client.?.objecter ms_handle_connect 0x7f083013e820 2024-03-29T11:03:00.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:00.640 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149049516 0 0) 0x7f083006eb70 con 0x7f083013e820 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f083006ee50 con 0x7f083013e820 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2544967151 0 0) 0x7f081c004eb0 con 0x7f083006d9d0 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f083006eb70 con 0x7f083006d9d0 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f08240017d0 con 0x7f083006d9d0 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): dump: 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:00.641 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 525510164 0 0) 0x7f083006ee50 con 0x7f083013e820 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f081c004eb0 con 0x7f083013e820 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0828001830 con 0x7f083013e820 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:00.642 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.658+0000 7f0817fff640 10 monclient(hunting): dump: 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.659+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 983430718 0 0) 0x7f083006eb70 con 0x7f083006d9d0 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.659+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 >> v1:172.21.15.26:6789/0 conn(0x7f083013e820 legacy=0x7f0830126950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:00.643 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.659+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 >> v1:172.21.15.22:6789/0 conn(0x7f08300af1d0 legacy=0x7f0830123220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 1 monclient: found mon.c 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083008abc0 con 0x7f083006d9d0 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 10 monclient: _finish_auth 0 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 5 monclient: authenticate success, global_id 4646 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 10 monclient: _renew_subs 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083006de40 con 0x7f083006d9d0 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 10 client.4646.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:00.644 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 10 monclient: _renew_subs 2024-03-29T11:03:00.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:00.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f083008b0e0 con 0x7f083006d9d0 2024-03-29T11:03:00.645 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.661+0000 7f0835d6b640 1 librados: init done 2024-03-29T11:03:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f08240019b0 con 0x7f083006d9d0 2024-03-29T11:03:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f0817fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f0824004a00 con 0x7f083006d9d0 2024-03-29T11:03:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f0817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f0817fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:00.646 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f0817fff640 10 monclient: dump: 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f082dffb640 4 set_mon_vals no callback set 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.662+0000 7f0835d6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.663+0000 7f0835d6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:00.647 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.663+0000 7f0835d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.663+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0810005080 con 0x7f083006d9d0 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.664+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f082401a9d0 con 0x7f083006d9d0 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.665+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f0824084a30 con 0x7f083006d9d0 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.665+0000 7f0817fff640 10 client.4646.objecter ms_dispatch 0x7f0830000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.665+0000 7f0817fff640 3 client.4646.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.665+0000 7f0817fff640 3 client.4646.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.665+0000 7f0817fff640 20 client.4646.objecter dump_active .. 0 homeless 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.665+0000 7f082effd640 10 monclient: get_auth_request con 0x7f081c06e030 auth_method 0 2024-03-29T11:03:00.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.666+0000 7f0817fff640 10 client.4646.objecter ms_handle_connect 0x7f081c06e030 2024-03-29T11:03:00.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.675+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0824050140 con 0x7f083006d9d0 2024-03-29T11:03:00.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.675+0000 7f0817fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:00.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:00.675+0000 7f0817fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:01.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.028+0000 7f0835d6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:01.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.028+0000 7f0835d6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:01.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.028+0000 7f0835d6b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:01.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.028+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0810005330 con 0x7f083006d9d0 2024-03-29T11:03:01.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.030+0000 7f0817fff640 1 -- 172.21.15.22:0/4120071167 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f0824052240 con 0x7f083006d9d0 2024-03-29T11:03:01.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.030+0000 7f0817fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:01.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.030+0000 7f0817fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:01.010 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:01.010 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:01.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.036+0000 7f0835d6b640 10 librados: watch_flush enter 2024-03-29T11:03:01.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.037+0000 7f0835d6b640 10 librados: watch_flush exit 2024-03-29T11:03:01.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.037+0000 7f0835d6b640 20 client.4646.objecter shutdown clearing up homeless session... 2024-03-29T11:03:01.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.037+0000 7f0835d6b640 10 client.4646.objecter successfully canceled tick 2024-03-29T11:03:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.038+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 >> v1:172.21.15.22:6816/380221644 conn(0x7f081c06e030 legacy=0x7f081c0704f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.038+0000 7f0835d6b640 10 monclient: shutdown 2024-03-29T11:03:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.038+0000 7f0835d6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.038+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 >> v1:172.21.15.22:6790/0 conn(0x7f083006d9d0 legacy=0x7f08300b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.038+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 shutdown_connections 2024-03-29T11:03:01.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.038+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 >> 172.21.15.22:0/4120071167 conn(0x7f083006b660 msgr2=0x7f083013ae40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:01.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.039+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 shutdown_connections 2024-03-29T11:03:01.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.039+0000 7f0835d6b640 1 -- 172.21.15.22:0/4120071167 wait complete. 2024-03-29T11:03:01.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:01.039+0000 7f0835d6b640 1 librados: shutdown 2024-03-29T11:03:01.039 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:07.846 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:07.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.018+0000 7f4b2056b640 10 monclient: get_monmap_and_config 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.018+0000 7f4b2056b640 10 monclient: build_initial_monmap 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.018+0000 7f4b2056b640 10 monclient: monmap: 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:08.019065+0000 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:08.019065+0000 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:07.998 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:07.999 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:07.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.019+0000 7f4b2056b640 1 Processor -- start 2024-03-29T11:03:07.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.019+0000 7f4b2056b640 1 -- start start 2024-03-29T11:03:07.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.019+0000 7f4b2056b640 10 monclient: init 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 10 monclient(hunting): picked mon.noname-b con 0x7f4b180b0770 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 10 monclient(hunting): picked mon.noname-a con 0x7f4b1814a700 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 10 monclient(hunting): picked mon.noname-c con 0x7f4b180ac5a0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180aba70 con 0x7f4b1814a700 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180aecb0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b180af150 con 0x7f4b180b0770 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b2056b640 10 monclient(hunting): authenticate will time out at 1289.324562s 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b1db06640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f4b1814a700 0x7f4b180abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48328/0 (socket says 172.21.15.22:48328) 2024-03-29T11:03:08.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.020+0000 7f4b1db06640 1 -- 172.21.15.22:0/1608595959 learned_addr learned my addr 172.21.15.22:0/1608595959 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:08.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398049190 0 0) 0x7f4b180af150 con 0x7f4b180b0770 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): _negotiate 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b04003520 con 0x7f4b180b0770 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180616940 0 0) 0x7f4b180aecb0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): _negotiate 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b180af150 con 0x7f4b180ac5a0 2024-03-29T11:03:08.002 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2508113008 0 0) 0x7f4b180af150 con 0x7f4b180ac5a0 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 >> v1:172.21.15.22:6790/0 conn(0x7f4b180b0770 legacy=0x7f4b18149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 >> v1:172.21.15.22:6789/0 conn(0x7f4b1814a700 legacy=0x7f4b180abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient: _finish_hunting 0 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 monclient: found mon.noname-c 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b180ac9d0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient: _finish_auth 0 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 10 monclient: _check_auth_tickets 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.021+0000 7f4b1d305640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.022+0000 7f4b2056b640 5 monclient: authenticate success, global_id 4474 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.022+0000 7f4b2056b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.022+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4b100019b0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.022+0000 7f4b1d305640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:08.003 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.022+0000 7f4b1d305640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.022+0000 7f4b1d305640 10 monclient: dump: 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:08.004 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.022+0000 7f4b2056b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:08.006 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b10001c40 con 0x7f4b180ac5a0 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b1d305640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b2056b640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b2056b640 4 set_mon_vals no callback set 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b1d305640 1 -- 172.21.15.22:0/1608595959 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4b10001fd0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b1d305640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b1d305640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b1d305640 10 monclient: dump: 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:08.007 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b2056b640 10 monclient: shutdown 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b2056b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.023+0000 7f4b2056b640 1 -- 172.21.15.22:0/1608595959 >> v1:172.21.15.26:6789/0 conn(0x7f4b180ac5a0 legacy=0x7f4b18151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.024+0000 7f4b2056b640 1 -- 172.21.15.22:0/1608595959 shutdown_connections 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.024+0000 7f4b2056b640 1 -- 172.21.15.22:0/1608595959 >> 172.21.15.22:0/1608595959 conn(0x7f4b1806b660 msgr2=0x7f4b1806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.024+0000 7f4b2056b640 1 -- 172.21.15.22:0/1608595959 shutdown_connections 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.024+0000 7f4b2056b640 1 -- 172.21.15.22:0/1608595959 wait complete. 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.025+0000 7f4b2056b640 10 monclient: build_initial_monmap 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.025+0000 7f4b2056b640 10 monclient: monmap: 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:08.008 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.025+0000 7f4b2056b640 1 librados: starting msgr at 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.025+0000 7f4b2056b640 1 librados: starting objecter 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.025+0000 7f4b2056b640 1 Processor -- start 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.026+0000 7f4b2056b640 1 -- start start 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.026+0000 7f4b2056b640 1 librados: setting wanted keys 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.026+0000 7f4b2056b640 1 librados: calling monclient init 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.026+0000 7f4b2056b640 10 monclient: init 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.026+0000 7f4b2056b640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:08.009 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.026+0000 7f4b2056b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 10 monclient(hunting): picked mon.noname-a con 0x7f4b180ac5a0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 10 monclient(hunting): picked mon.noname-b con 0x7f4b180b0770 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 10 monclient(hunting): picked mon.noname-c con 0x7f4b1814a700 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b18129ec0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b1812a110 con 0x7f4b180b0770 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b18129350 con 0x7f4b1814a700 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b2056b640 10 monclient(hunting): authenticate will time out at 1289.331289s 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b1e307640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f4b180ac5a0 0x7f4b18146d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:48586/0 (socket says 172.21.15.22:48586) 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b1e307640 1 -- 172.21.15.22:0/1272569195 learned_addr learned my addr 172.21.15.22:0/1272569195 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b0effd640 10 client.?.objecter ms_handle_connect 0x7f4b180ac5a0 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b0effd640 10 client.?.objecter ms_handle_connect 0x7f4b1814a700 2024-03-29T11:03:08.010 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.027+0000 7f4b0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090034903 0 0) 0x7f4b18129ec0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): my global_id is 4500 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4afc004eb0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596965227 0 0) 0x7f4b18129350 con 0x7f4b1814a700 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b18129ec0 con 0x7f4b1814a700 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 client.?.objecter ms_handle_connect 0x7f4b180b0770 2024-03-29T11:03:08.015 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b1eb08640 1 -- 172.21.15.22:0/1272569195 >> v1:172.21.15.22:6790/0 conn(0x7f4b1814a700 legacy=0x7f4b181df4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b1eb08640 1 -- 172.21.15.22:0/1272569195 >> v1:172.21.15.22:6790/0 conn(0x7f4b1814a700 legacy=0x7f4b181df4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b1eb08640 1 --1- 172.21.15.22:0/1272569195 >> v1:172.21.15.22:6790/0 conn(0x7f4b1814a700 0x7f4b181df4b0 :-1 s=OPENED pgs=935681 cs=1 l=1).handle_message read tag failed 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b1eb08640 1 --1- 172.21.15.22:0/1272569195 >> v1:172.21.15.22:6790/0 conn(0x7f4b1814a700 0x7f4b181df4b0 :-1 s=OPENED pgs=935681 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.22:6790/0 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713392193 0 0) 0x7f4b1812a110 con 0x7f4b180b0770 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.028+0000 7f4b0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b18129350 con 0x7f4b180b0770 2024-03-29T11:03:08.016 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3491680781 0 0) 0x7f4afc004eb0 con 0x7f4b180ac5a0 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4b1812a110 con 0x7f4b180ac5a0 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4b08001990 con 0x7f4b180ac5a0 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 10 monclient(hunting): dump: 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:08.017 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3011166994 0 0) 0x7f4b18129350 con 0x7f4b180b0770 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4afc004eb0 con 0x7f4b180b0770 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.029+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4b14001790 con 0x7f4b180b0770 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 10 monclient(hunting): dump: 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:08.018 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 449141696 0 0) 0x7f4afc004eb0 con 0x7f4b180b0770 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 >> v1:172.21.15.22:6790/0 conn(0x7f4b1814a700 legacy=0x7f4b181df4b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 >> v1:172.21.15.22:6789/0 conn(0x7f4b180ac5a0 legacy=0x7f4b18146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 10 monclient: _finish_hunting 0 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 1 monclient: found mon.b 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18147a10 con 0x7f4b180b0770 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 10 monclient: _finish_auth 0 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 10 monclient: _check_auth_tickets 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:03:08.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 10 monclient: _renew_subs 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b18128000 con 0x7f4b180b0770 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 10 monclient: _renew_subs 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b18128e50 con 0x7f4b180b0770 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.030+0000 7f4b2056b640 1 librados: init done 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.031+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4b14001ba0 con 0x7f4b180b0770 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.031+0000 7f4b0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.031+0000 7f4b1cb04640 4 set_mon_vals no callback set 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.031+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4b14004980 con 0x7f4b180b0770 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.031+0000 7f4b0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.031+0000 7f4b0effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:08.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.031+0000 7f4b0effd640 10 monclient: dump: 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f4b14004ae0 con 0x7f4b180b0770 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b2056b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:08.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b2056b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b2056b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f4b14084890 con 0x7f4b180b0770 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b0effd640 10 client.4477.objecter ms_dispatch 0x7f4b18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b0effd640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b0effd640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b0effd640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.033+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b00005080 con 0x7f4b180b0770 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.034+0000 7f4b1e307640 10 monclient: get_auth_request con 0x7f4afc06dfe0 auth_method 0 2024-03-29T11:03:08.022 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.034+0000 7f4b0effd640 10 client.4477.objecter ms_handle_connect 0x7f4afc06dfe0 2024-03-29T11:03:08.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.043+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4b1404ffa0 con 0x7f4b180b0770 2024-03-29T11:03:08.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.043+0000 7f4b0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:08.023 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.043+0000 7f4b0effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:08.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.402+0000 7f4b2056b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:08.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.402+0000 7f4b2056b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:08.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.402+0000 7f4b2056b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:08.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.402+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b00005330 con 0x7f4b180b0770 2024-03-29T11:03:08.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.403+0000 7f4b0effd640 1 -- 172.21.15.22:0/1272569195 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f4b140520a0 con 0x7f4b180b0770 2024-03-29T11:03:08.382 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.403+0000 7f4b0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:08.383 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.403+0000 7f4b0effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:08.383 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:08.383 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:08.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 10 librados: watch_flush enter 2024-03-29T11:03:08.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 10 librados: watch_flush exit 2024-03-29T11:03:08.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:03:08.386 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 >> v1:172.21.15.22:6816/380221644 conn(0x7f4afc06dfe0 legacy=0x7f4afc0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 10 monclient: shutdown 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 >> v1:172.21.15.26:6789/0 conn(0x7f4b180b0770 legacy=0x7f4b181deda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 shutdown_connections 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 >> 172.21.15.22:0/1272569195 conn(0x7f4b1806b660 msgr2=0x7f4b18148490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 shutdown_connections 2024-03-29T11:03:08.387 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.407+0000 7f4b2056b640 1 -- 172.21.15.22:0/1272569195 wait complete. 2024-03-29T11:03:08.388 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:08.408+0000 7f4b2056b640 1 librados: shutdown 2024-03-29T11:03:08.406 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:15.213 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:15.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.367+0000 7ff7e0140640 10 monclient: get_monmap_and_config 2024-03-29T11:03:15.346 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.367+0000 7ff7e0140640 10 monclient: build_initial_monmap 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.367+0000 7ff7e0140640 10 monclient: monmap: 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:15.367841+0000 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:15.367841+0000 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:15.347 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.367+0000 7ff7e0140640 1 Processor -- start 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.368+0000 7ff7e0140640 1 -- start start 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.368+0000 7ff7e0140640 10 monclient: init 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.368+0000 7ff7e0140640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.368+0000 7ff7e0140640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.368+0000 7ff7e0140640 10 monclient(hunting): picked mon.noname-a con 0x7ff7d81457f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7e0140640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d813f650 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7e0140640 10 monclient(hunting): picked mon.noname-b con 0x7ff7d80a9d70 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7e0140640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d80aaef0 con 0x7ff7d81457f0 2024-03-29T11:03:15.350 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7e0140640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d8145bd0 con 0x7ff7d813f650 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7e0140640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d8145eb0 con 0x7ff7d80a9d70 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7e0140640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7e0140640 10 monclient(hunting): authenticate will time out at 1296.673212s 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7de6dd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff7d80a9d70 0x7ff7d814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:54578/0 (socket says 172.21.15.22:54578) 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7de6dd640 1 -- 172.21.15.22:0/294179766 learned_addr learned my addr 172.21.15.22:0/294179766 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159461997 0 0) 0x7ff7d8145eb0 con 0x7ff7d80a9d70 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): _negotiate 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7c4003520 con 0x7ff7d80a9d70 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469394267 0 0) 0x7ff7d8145bd0 con 0x7ff7d813f650 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): _negotiate 2024-03-29T11:03:15.351 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:15.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:15.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:03:15.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.369+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7d8145eb0 con 0x7ff7d813f650 2024-03-29T11:03:15.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.370+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009081447 0 0) 0x7ff7d8145eb0 con 0x7ff7d813f650 2024-03-29T11:03:15.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.370+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 >> v1:172.21.15.22:6790/0 conn(0x7ff7d80a9d70 legacy=0x7ff7d814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:15.352 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.370+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 >> v1:172.21.15.22:6789/0 conn(0x7ff7d81457f0 legacy=0x7ff7d813ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 10 monclient: _finish_hunting 0 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 1 monclient: found mon.noname-c 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d8146340 con 0x7ff7d813f650 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 10 monclient: _finish_auth 0 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 10 monclient: _check_auth_tickets 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff7d00019f0 con 0x7ff7d813f650 2024-03-29T11:03:15.353 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7dceda640 10 monclient: dump: 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:15.354 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7e0140640 5 monclient: authenticate success, global_id 4480 2024-03-29T11:03:15.355 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.372+0000 7ff7e0140640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:15.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7d0001c10 con 0x7ff7d813f650 2024-03-29T11:03:15.356 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7dceda640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7dceda640 1 -- 172.21.15.22:0/294179766 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff7d0001fa0 con 0x7ff7d813f650 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7dceda640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7dceda640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7dceda640 10 monclient: dump: 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:15.357 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7e0140640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.374+0000 7ff7e0140640 4 set_mon_vals no callback set 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.375+0000 7ff7e0140640 10 monclient: shutdown 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.375+0000 7ff7e0140640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.375+0000 7ff7e0140640 1 -- 172.21.15.22:0/294179766 >> v1:172.21.15.26:6789/0 conn(0x7ff7d813f650 legacy=0x7ff7d80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.376+0000 7ff7e0140640 1 -- 172.21.15.22:0/294179766 shutdown_connections 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.376+0000 7ff7e0140640 1 -- 172.21.15.22:0/294179766 >> 172.21.15.22:0/294179766 conn(0x7ff7d8066a00 msgr2=0x7ff7d8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.376+0000 7ff7e0140640 1 -- 172.21.15.22:0/294179766 shutdown_connections 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.376+0000 7ff7e0140640 1 -- 172.21.15.22:0/294179766 wait complete. 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.377+0000 7ff7e0140640 10 monclient: build_initial_monmap 2024-03-29T11:03:15.358 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.377+0000 7ff7e0140640 10 monclient: monmap: 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:15.359 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:15.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.378+0000 7ff7e0140640 1 librados: starting msgr at 2024-03-29T11:03:15.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.378+0000 7ff7e0140640 1 librados: starting objecter 2024-03-29T11:03:15.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.378+0000 7ff7e0140640 1 Processor -- start 2024-03-29T11:03:15.362 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.378+0000 7ff7e0140640 1 -- start start 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.378+0000 7ff7e0140640 1 librados: setting wanted keys 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.378+0000 7ff7e0140640 1 librados: calling monclient init 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.378+0000 7ff7e0140640 10 monclient: init 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 10 monclient(hunting): picked mon.noname-b con 0x7ff7d80a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 10 monclient(hunting): picked mon.noname-a con 0x7ff7d813f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 10 monclient(hunting): picked mon.noname-c con 0x7ff7d81457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d8141dd0 con 0x7ff7d813f650 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d81420b0 con 0x7ff7d80a9d70 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7d813b6e0 con 0x7ff7d81457f0 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:15.363 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.379+0000 7ff7e0140640 10 monclient(hunting): authenticate will time out at 1296.683688s 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7de6dd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7ff7d81457f0 0x7ff7d81e64c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55596/0 (socket says 172.21.15.22:55596) 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7de6dd640 1 -- 172.21.15.22:0/725973070 learned_addr learned my addr 172.21.15.22:0/725973070 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 10 client.?.objecter ms_handle_connect 0x7ff7d81457f0 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739958136 0 0) 0x7ff7d813b6e0 con 0x7ff7d81457f0 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.380+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7bc004eb0 con 0x7ff7d81457f0 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 555931496 0 0) 0x7ff7bc004eb0 con 0x7ff7d81457f0 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7d813b6e0 con 0x7ff7d81457f0 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff7d4001950 con 0x7ff7d81457f0 2024-03-29T11:03:15.364 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient(hunting): dump: 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1512507825 0 0) 0x7ff7d813b6e0 con 0x7ff7d81457f0 2024-03-29T11:03:15.365 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 >> v1:172.21.15.26:6789/0 conn(0x7ff7d80a9d70 legacy=0x7ff7d81424d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 >> v1:172.21.15.22:6789/0 conn(0x7ff7d813f650 legacy=0x7ff7d8139e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 monclient: found mon.c 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7d8142be0 con 0x7ff7d81457f0 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7e0140640 5 monclient: authenticate success, global_id 4658 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7e0140640 10 monclient: _renew_subs 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7e0140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7d8142e50 con 0x7ff7d81457f0 2024-03-29T11:03:15.366 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7e0140640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7e0140640 10 monclient: _renew_subs 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.381+0000 7ff7e0140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7d8140f50 con 0x7ff7d81457f0 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7e0140640 1 librados: init done 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7d40047e0 con 0x7ff7d81457f0 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7ff7d4004c30 con 0x7ff7d81457f0 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7ce7fc640 10 monclient: dump: 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:15.367 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:15.368 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:15.368 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:15.368 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:15.368 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:15.368 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:15.368 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.382+0000 7ff7cffff640 4 set_mon_vals no callback set 2024-03-29T11:03:15.369 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.384+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7ff7d4004e10 con 0x7ff7d81457f0 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.384+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7ff7d4084b80 con 0x7ff7d81457f0 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.384+0000 7ff7ce7fc640 10 client.4658.objecter ms_dispatch 0x7ff7d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.384+0000 7ff7ce7fc640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.384+0000 7ff7ce7fc640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.385+0000 7ff7ce7fc640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.385+0000 7ff7ddedc640 10 monclient: get_auth_request con 0x7ff7bc06dbc0 auth_method 0 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.385+0000 7ff7ce7fc640 10 client.4658.objecter ms_handle_connect 0x7ff7bc06dbc0 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.385+0000 7ff7e0140640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.385+0000 7ff7e0140640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.385+0000 7ff7e0140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:15.370 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.385+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c0005080 con 0x7ff7d81457f0 2024-03-29T11:03:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.394+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7d4050460 con 0x7ff7d81457f0 2024-03-29T11:03:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.394+0000 7ff7ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:15.373 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.394+0000 7ff7ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:15.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.712+0000 7ff7e0140640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:15.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.712+0000 7ff7e0140640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:15.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.712+0000 7ff7e0140640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:15.691 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.712+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff7c0005330 con 0x7ff7d81457f0 2024-03-29T11:03:15.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.712+0000 7ff7ce7fc640 1 -- 172.21.15.22:0/725973070 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7ff7d4052390 con 0x7ff7d81457f0 2024-03-29T11:03:15.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.712+0000 7ff7ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:15.692 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.712+0000 7ff7ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:15.692 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:15.692 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:15.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 10 librados: watch_flush enter 2024-03-29T11:03:15.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 10 librados: watch_flush exit 2024-03-29T11:03:15.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-29T11:03:15.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 10 client.4658.objecter successfully canceled tick 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 >> v1:172.21.15.22:6816/380221644 conn(0x7ff7bc06dbc0 legacy=0x7ff7bc070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 10 monclient: shutdown 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.715+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 >> v1:172.21.15.22:6790/0 conn(0x7ff7d81457f0 legacy=0x7ff7d81e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.716+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 shutdown_connections 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.716+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 >> 172.21.15.22:0/725973070 conn(0x7ff7d8066a00 msgr2=0x7ff7d80b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.716+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 shutdown_connections 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.716+0000 7ff7e0140640 1 -- 172.21.15.22:0/725973070 wait complete. 2024-03-29T11:03:15.696 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:15.716+0000 7ff7e0140640 1 librados: shutdown 2024-03-29T11:03:15.710 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:22.517 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:22.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.677+0000 7f84e1615640 10 monclient: get_monmap_and_config 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.677+0000 7f84e1615640 10 monclient: build_initial_monmap 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.677+0000 7f84e1615640 10 monclient: monmap: 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:22.678070+0000 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:22.678070+0000 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:22.657 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:22.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.678+0000 7f84e1615640 1 Processor -- start 2024-03-29T11:03:22.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.678+0000 7f84e1615640 1 -- start start 2024-03-29T11:03:22.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.678+0000 7f84e1615640 10 monclient: init 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84e1615640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84e1615640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84e1615640 10 monclient(hunting): picked mon.noname-b con 0x7f84dc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84daffd640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f84dc1457f0 legacy=0x7f84dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84daffd640 1 -- >> v1:172.21.15.22:6790/0 conn(0x7f84dc1457f0 legacy=0x7f84dc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84e1615640 10 monclient(hunting): picked mon.noname-c con 0x7f84dc13f650 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84e1615640 10 monclient(hunting): picked mon.noname-a con 0x7f84dc0a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.679+0000 7f84e1615640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc0aaef0 con 0x7f84dc0a9d70 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.680+0000 7f84e1615640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc145bd0 con 0x7f84dc13f650 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.680+0000 7f84e1615640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc145eb0 con 0x7f84dc1457f0 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.680+0000 7f84e1615640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.680+0000 7f84e1615640 10 monclient(hunting): authenticate will time out at 1303.984144s 2024-03-29T11:03:22.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.680+0000 7f84daffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f84dc1457f0 0x7f84dc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify read banner and identify addresses failed 2024-03-29T11:03:22.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.680+0000 7f84d3fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f84dc13f650 0x7f84dc0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:51548/0 (socket says 172.21.15.22:51548) 2024-03-29T11:03:22.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.680+0000 7f84d3fff640 1 -- 172.21.15.22:0/138903791 learned_addr learned my addr 172.21.15.22:0/138903791 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:22.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.681+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2600640108 0 0) 0x7f84dc145bd0 con 0x7f84dc13f650 2024-03-29T11:03:22.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.681+0000 7f84da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:22.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.681+0000 7f84da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:22.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.681+0000 7f84da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:22.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.681+0000 7f84da7fc640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:03:22.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.681+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c8003520 con 0x7f84dc13f650 2024-03-29T11:03:22.667 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168175362 0 0) 0x7f84dc0aaef0 con 0x7f84dc0a9d70 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient(hunting): my global_id is 4503 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84dc145bd0 con 0x7f84dc0a9d70 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3323751415 0 0) 0x7f84c8003520 con 0x7f84dc13f650 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 >> v1:172.21.15.22:6790/0 conn(0x7f84dc1457f0 legacy=0x7f84dc13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 >> v1:172.21.15.22:6789/0 conn(0x7f84dc0a9d70 legacy=0x7f84dc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 monclient: found mon.noname-c 2024-03-29T11:03:22.668 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc146340 con 0x7f84dc13f650 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f84c4001610 con 0x7f84dc13f650 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84da7fc640 10 monclient: dump: 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.669 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84e1615640 5 monclient: authenticate success, global_id 4483 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.684+0000 7f84e1615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84c4001770 con 0x7f84dc13f650 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84da7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84da7fc640 1 -- 172.21.15.22:0/138903791 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f84c4001bc0 con 0x7f84dc13f650 2024-03-29T11:03:22.670 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84da7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84da7fc640 10 monclient: dump: 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84e1615640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:22.671 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.685+0000 7f84e1615640 4 set_mon_vals no callback set 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.686+0000 7f84e1615640 10 monclient: shutdown 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.686+0000 7f84e1615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.686+0000 7f84e1615640 1 -- 172.21.15.22:0/138903791 >> v1:172.21.15.26:6789/0 conn(0x7f84dc13f650 legacy=0x7f84dc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.686+0000 7f84e1615640 1 -- 172.21.15.22:0/138903791 shutdown_connections 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.686+0000 7f84e1615640 1 -- 172.21.15.22:0/138903791 >> 172.21.15.22:0/138903791 conn(0x7f84dc066a00 msgr2=0x7f84dc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.686+0000 7f84e1615640 1 -- 172.21.15.22:0/138903791 shutdown_connections 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.686+0000 7f84e1615640 1 -- 172.21.15.22:0/138903791 wait complete. 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.687+0000 7f84e1615640 10 monclient: build_initial_monmap 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.687+0000 7f84e1615640 10 monclient: monmap: 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:22.672 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.687+0000 7f84e1615640 1 librados: starting msgr at 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.687+0000 7f84e1615640 1 librados: starting objecter 2024-03-29T11:03:22.673 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.688+0000 7f84e1615640 1 Processor -- start 2024-03-29T11:03:22.678 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.688+0000 7f84e1615640 1 -- start start 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.688+0000 7f84e1615640 1 librados: setting wanted keys 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.688+0000 7f84e1615640 1 librados: calling monclient init 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.688+0000 7f84e1615640 10 monclient: init 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.689+0000 7f84e1615640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.689+0000 7f84e1615640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.689+0000 7f84e1615640 10 monclient(hunting): picked mon.noname-b con 0x7f84dc0a9d70 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.689+0000 7f84daffd640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f84dc0a9d70 0x7f84dc1d76f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:52390/0 (socket says 172.21.15.22:52390) 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84e1615640 10 monclient(hunting): picked mon.noname-a con 0x7f84dc13f650 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84daffd640 1 -- 172.21.15.22:0/4240506999 learned_addr learned my addr 172.21.15.22:0/4240506999 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84e1615640 10 monclient(hunting): picked mon.noname-c con 0x7f84dc1457f0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84e1615640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc0ac6c0 con 0x7f84dc13f650 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84e1615640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc137820 con 0x7f84dc0a9d70 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84e1615640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc1429e0 con 0x7f84dc1457f0 2024-03-29T11:03:22.679 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84e1615640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84e1615640 10 monclient(hunting): authenticate will time out at 1303.995272s 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 client.?.objecter ms_handle_connect 0x7f84dc1457f0 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060941178 0 0) 0x7f84dc1429e0 con 0x7f84dc1457f0 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84bc004eb0 con 0x7f84dc1457f0 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3389171200 0 0) 0x7f84bc004eb0 con 0x7f84dc1457f0 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84dc1429e0 con 0x7f84dc1457f0 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f84d4001cb0 con 0x7f84dc1457f0 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:22.680 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): dump: 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 546082329 0 0) 0x7f84dc1429e0 con 0x7f84dc1457f0 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 >> v1:172.21.15.26:6789/0 conn(0x7f84dc0a9d70 legacy=0x7f84dc1d76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:22.681 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.691+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 >> v1:172.21.15.22:6789/0 conn(0x7f84dc13f650 legacy=0x7f84dc136a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 1 monclient: found mon.c 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc143080 con 0x7f84dc1457f0 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: _finish_auth 0 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 10 monclient: _renew_subs 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc1426b0 con 0x7f84dc1457f0 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 10 monclient: _renew_subs 2024-03-29T11:03:22.682 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc141ed0 con 0x7f84dc1457f0 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84e1615640 1 librados: init done 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84d4001e10 con 0x7f84dc1457f0 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f84d4003c40 con 0x7f84dc1457f0 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d37fe640 10 monclient: dump: 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:22.683 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.692+0000 7f84d9ffb640 4 set_mon_vals no callback set 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.693+0000 7f84d17fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.693+0000 7f84d17fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.693+0000 7f84d17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.693+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc075f10 con 0x7f84dc1457f0 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.694+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f84d4019c10 con 0x7f84dc1457f0 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.695+0000 7f84daffd640 10 monclient: get_auth_request con 0x7f84bc076410 auth_method 0 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.695+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f84d4083c30 con 0x7f84dc1457f0 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.695+0000 7f84d37fe640 10 client.4661.objecter ms_dispatch 0x7f84dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:22.684 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.695+0000 7f84d37fe640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:22.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.695+0000 7f84d37fe640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:22.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.695+0000 7f84d37fe640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:03:22.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.695+0000 7f84d37fe640 10 client.4661.objecter ms_handle_connect 0x7f84bc076410 2024-03-29T11:03:22.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.701+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84d404f340 con 0x7f84dc1457f0 2024-03-29T11:03:22.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.701+0000 7f84d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:22.685 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.701+0000 7f84d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:22.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.995+0000 7f84d17fa640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:22.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.995+0000 7f84d17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:22.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.995+0000 7f84d17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:22.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.995+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84dc0b66e0 con 0x7f84dc1457f0 2024-03-29T11:03:22.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.995+0000 7f84d37fe640 1 -- 172.21.15.22:0/4240506999 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f84d4051440 con 0x7f84dc1457f0 2024-03-29T11:03:22.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.995+0000 7f84d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:22.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.995+0000 7f84d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:22.976 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:22.976 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:22.977 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.997+0000 7f84d17fa640 10 librados: watch_flush enter 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.997+0000 7f84d17fa640 10 librados: watch_flush exit 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.997+0000 7f84d17fa640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.997+0000 7f84d17fa640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 >> v1:172.21.15.22:6816/380221644 conn(0x7f84bc076410 legacy=0x7f84bc0788d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 10 monclient: shutdown 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 >> v1:172.21.15.22:6790/0 conn(0x7f84dc1457f0 legacy=0x7f84dc137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 shutdown_connections 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 >> 172.21.15.22:0/4240506999 conn(0x7f84dc066a00 msgr2=0x7f84dc0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 shutdown_connections 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 1 -- 172.21.15.22:0/4240506999 wait complete. 2024-03-29T11:03:22.978 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:22.998+0000 7f84d17fa640 1 librados: shutdown 2024-03-29T11:03:22.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:29.797 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:29.949 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.970+0000 7f73cf59e640 10 monclient: get_monmap_and_config 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.970+0000 7f73cf59e640 10 monclient: build_initial_monmap 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.970+0000 7f73cf59e640 10 monclient: monmap: 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:29.971189+0000 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:29.971189+0000 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:29.950 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:29.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.971+0000 7f73cf59e640 1 Processor -- start 2024-03-29T11:03:29.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.971+0000 7f73cf59e640 1 -- start start 2024-03-29T11:03:29.951 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.971+0000 7f73cf59e640 10 monclient: init 2024-03-29T11:03:29.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:29.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:29.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f73c80389e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:29.952 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f73c8038dc0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f73c8030660 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c80c54c0 con 0x7f73c8038dc0 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c8036430 con 0x7f73c8030660 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c8039840 con 0x7f73c80389e0 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.972+0000 7f73cf59e640 10 monclient(hunting): authenticate will time out at 1311.276687s 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.973+0000 7f73ce59c640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f73c80389e0 0x7f73c8033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50118/0 (socket says 172.21.15.22:50118) 2024-03-29T11:03:29.953 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.973+0000 7f73ce59c640 1 -- 172.21.15.22:0/851427627 learned_addr learned my addr 172.21.15.22:0/851427627 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:29.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.974+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050119720 0 0) 0x7f73c8039840 con 0x7f73c80389e0 2024-03-29T11:03:29.954 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.974+0000 7f73cd59a640 10 monclient(hunting): _negotiate 2024-03-29T11:03:29.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.974+0000 7f73cd59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:29.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.974+0000 7f73cd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:29.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.974+0000 7f73cd59a640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:03:29.955 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.974+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b8003520 con 0x7f73c80389e0 2024-03-29T11:03:29.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 267401375 0 0) 0x7f73b8003520 con 0x7f73c80389e0 2024-03-29T11:03:29.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 >> v1:172.21.15.26:6789/0 conn(0x7f73c8030660 legacy=0x7f73c80d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:03:29.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 >> v1:172.21.15.22:6789/0 conn(0x7f73c8038dc0 legacy=0x7f73c80cadd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:29.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: _finish_hunting 0 2024-03-29T11:03:29.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 1 monclient: found mon.noname-b 2024-03-29T11:03:29.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:29.957 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c8035690 con 0x7f73c80389e0 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: _finish_auth 0 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: _check_auth_tickets 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f73c4001690 con 0x7f73c80389e0 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: dump: 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:29.958 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cf59e640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cf59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73c4001860 con 0x7f73c80389e0 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cf59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.976+0000 7f73cf59e640 4 set_mon_vals no callback set 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cd59a640 1 -- 172.21.15.22:0/851427627 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f73c4001cb0 con 0x7f73c80389e0 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cd59a640 10 monclient: dump: 2024-03-29T11:03:29.959 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cf59e640 10 monclient: shutdown 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cf59e640 1 -- 172.21.15.22:0/851427627 >> v1:172.21.15.22:6790/0 conn(0x7f73c80389e0 legacy=0x7f73c8033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.977+0000 7f73cf59e640 1 -- 172.21.15.22:0/851427627 shutdown_connections 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.978+0000 7f73cf59e640 1 -- 172.21.15.22:0/851427627 >> 172.21.15.22:0/851427627 conn(0x7f73c8005c00 msgr2=0x7f73c8006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:29.960 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.978+0000 7f73cf59e640 1 -- 172.21.15.22:0/851427627 shutdown_connections 2024-03-29T11:03:29.961 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.978+0000 7f73cf59e640 1 -- 172.21.15.22:0/851427627 wait complete. 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 10 monclient: build_initial_monmap 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 10 monclient: monmap: 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:29.963 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 1 librados: starting msgr at 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 1 librados: starting objecter 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 1 Processor -- start 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 1 -- start start 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 1 librados: setting wanted keys 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 1 librados: calling monclient init 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.979+0000 7f73cf59e640 10 monclient: init 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.980+0000 7f73cf59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.980+0000 7f73cf59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.980+0000 7f73cf59e640 10 monclient(hunting): picked mon.noname-a con 0x7f73c8030660 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cf59e640 10 monclient(hunting): picked mon.noname-c con 0x7f73c80389e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cf59e640 10 monclient(hunting): picked mon.noname-b con 0x7f73c80c82a0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cf59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c802f3a0 con 0x7f73c8030660 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cf59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c802ec30 con 0x7f73c80c82a0 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cf59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73c802ef10 con 0x7f73c80389e0 2024-03-29T11:03:29.964 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cf59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cf59e640 10 monclient(hunting): authenticate will time out at 1311.286179s 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cdd9b640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f73c80389e0 0x7f73c802f880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:50592/0 (socket says 172.21.15.22:50592) 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73cdd9b640 1 -- 172.21.15.22:0/3488887140 learned_addr learned my addr 172.21.15.22:0/3488887140 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73beffd640 10 client.?.objecter ms_handle_connect 0x7f73c80389e0 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.982+0000 7f73beffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.983+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614903404 0 0) 0x7f73c802ef10 con 0x7f73c80389e0 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.983+0000 7f73beffd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.983+0000 7f73beffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.983+0000 7f73beffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.983+0000 7f73beffd640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:03:29.965 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b0004eb0 con 0x7f73c80389e0 2024-03-29T11:03:29.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4291562419 0 0) 0x7f73b0004eb0 con 0x7f73c80389e0 2024-03-29T11:03:29.966 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73c802ef10 con 0x7f73c80389e0 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f73d004fdd0 con 0x7f73c80389e0 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 10 monclient(hunting): dump: 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:29.967 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:29.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.984+0000 7f73beffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:29.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.985+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 392264561 0 0) 0x7f73c802ef10 con 0x7f73c80389e0 2024-03-29T11:03:29.968 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.985+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 >> v1:172.21.15.26:6789/0 conn(0x7f73c80c82a0 legacy=0x7f73c8167320 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.988+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 >> v1:172.21.15.22:6789/0 conn(0x7f73c8030660 legacy=0x7f73c802d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 10 monclient: _finish_hunting 0 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 1 monclient: found mon.c 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73c802ff90 con 0x7f73c80389e0 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 10 monclient: _finish_auth 0 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 10 monclient: _check_auth_tickets 2024-03-29T11:03:29.969 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73beffd640 10 client.?.objecter ms_handle_connect 0x7f73c8030660 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 5 monclient: authenticate success, global_id 4667 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 10 monclient: _renew_subs 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73c8168a40 con 0x7f73c80389e0 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 10 monclient: _renew_subs 2024-03-29T11:03:29.970 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:29.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73c8168db0 con 0x7f73c80389e0 2024-03-29T11:03:29.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.989+0000 7f73cf59e640 1 librados: init done 2024-03-29T11:03:29.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.990+0000 7f73beffd640 10 client.4667.objecter resend_mon_ops 2024-03-29T11:03:29.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.991+0000 7f73cf59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:29.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.991+0000 7f73cf59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:29.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.991+0000 7f73cf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:29.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.991+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73c8007b90 con 0x7f73c80389e0 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.991+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73d004f400 con 0x7f73c80389e0 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.991+0000 7f73beffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.991+0000 7f73ccd99640 4 set_mon_vals no callback set 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.992+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f73d0075f10 con 0x7f73c80389e0 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.992+0000 7f73beffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.992+0000 7f73beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.992+0000 7f73beffd640 10 monclient: dump: 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.972 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:29.973 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:29.973 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:29.973 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:29.973 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:29.973 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:29.973 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.993+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f73d0067bf0 con 0x7f73c80389e0 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.994+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f73d00ff4f0 con 0x7f73c80389e0 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.994+0000 7f73beffd640 10 client.4667.objecter ms_dispatch 0x7f73d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.994+0000 7f73beffd640 3 client.4667.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.994+0000 7f73beffd640 3 client.4667.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.994+0000 7f73beffd640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.994+0000 7f73ce59c640 10 monclient: get_auth_request con 0x7f73b006daf0 auth_method 0 2024-03-29T11:03:29.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:29.994+0000 7f73beffd640 10 client.4667.objecter ms_handle_connect 0x7f73b006daf0 2024-03-29T11:03:29.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.000+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73d00cae30 con 0x7f73c80389e0 2024-03-29T11:03:29.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.000+0000 7f73beffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:29.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.000+0000 7f73beffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:30.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.334+0000 7f73cf59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:30.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.334+0000 7f73cf59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:30.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.334+0000 7f73cf59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:30.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.334+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73c803c490 con 0x7f73c80389e0 2024-03-29T11:03:30.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.336+0000 7f73beffd640 1 -- 172.21.15.22:0/3488887140 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f73d00ccd60 con 0x7f73c80389e0 2024-03-29T11:03:30.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.336+0000 7f73beffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:30.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.336+0000 7f73beffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:30.315 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:30.316 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:30.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 10 librados: watch_flush enter 2024-03-29T11:03:30.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 10 librados: watch_flush exit 2024-03-29T11:03:30.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-29T11:03:30.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 10 client.4667.objecter successfully canceled tick 2024-03-29T11:03:30.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 >> v1:172.21.15.22:6816/380221644 conn(0x7f73b006daf0 legacy=0x7f73b006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:30.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 10 monclient: shutdown 2024-03-29T11:03:30.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:30.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 >> v1:172.21.15.22:6790/0 conn(0x7f73c80389e0 legacy=0x7f73c802f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:30.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.339+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 shutdown_connections 2024-03-29T11:03:30.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.340+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 >> 172.21.15.22:0/3488887140 conn(0x7f73c8005c00 msgr2=0x7f73c8034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:30.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.340+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 shutdown_connections 2024-03-29T11:03:30.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.340+0000 7f73cf59e640 1 -- 172.21.15.22:0/3488887140 wait complete. 2024-03-29T11:03:30.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:30.340+0000 7f73cf59e640 1 librados: shutdown 2024-03-29T11:03:30.336 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:37.142 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:37.291 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.311+0000 7fa60e0a8640 10 monclient: get_monmap_and_config 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.311+0000 7fa60e0a8640 10 monclient: build_initial_monmap 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.311+0000 7fa60e0a8640 10 monclient: monmap: 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:37.313073+0000 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:37.313073+0000 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:37.292 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:37.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.312+0000 7fa60e0a8640 1 Processor -- start 2024-03-29T11:03:37.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.312+0000 7fa60e0a8640 1 -- start start 2024-03-29T11:03:37.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.312+0000 7fa60e0a8640 10 monclient: init 2024-03-29T11:03:37.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.313+0000 7fa60e0a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:37.293 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.313+0000 7fa60e0a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:37.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa6080743e0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:37.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6080747c0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:37.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa60813e680 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:37.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa60806f2e0 con 0x7fa60813e680 2024-03-29T11:03:37.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa608070b40 con 0x7fa6080743e0 2024-03-29T11:03:37.297 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa608070e20 con 0x7fa6080747c0 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa60e0a8640 10 monclient(hunting): authenticate will time out at 1318.619233s 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa606ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa6080747c0 0x7fa60813b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:40958/0 (socket says 172.21.15.22:40958) 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa606ffd640 1 -- 172.21.15.22:0/3367392509 learned_addr learned my addr 172.21.15.22:0/3367392509 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218268773 0 0) 0x7fa608070e20 con 0x7fa6080747c0 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f0003520 con 0x7fa6080747c0 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2441882269 0 0) 0x7fa5f0003520 con 0x7fa6080747c0 2024-03-29T11:03:37.298 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.314+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 >> v1:172.21.15.26:6789/0 conn(0x7fa6080743e0 legacy=0x7fa60806ebd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:37.299 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.317+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 >> v1:172.21.15.22:6789/0 conn(0x7fa60813e680 legacy=0x7fa60813eaf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 1 monclient: found mon.noname-b 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa608071100 con 0x7fa6080747c0 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:37.300 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa5fc001690 con 0x7fa6080747c0 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: dump: 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:37.302 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa60e0a8640 5 monclient: authenticate success, global_id 4670 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa60e0a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5fc001860 con 0x7fa6080747c0 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 1 -- 172.21.15.22:0/3367392509 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa5fc001cb0 con 0x7fa6080747c0 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.319+0000 7fa6067fc640 10 monclient: dump: 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:37.303 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:37.304 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:37.304 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:37.304 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:37.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.320+0000 7fa60e0a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:37.304 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.320+0000 7fa60e0a8640 4 set_mon_vals no callback set 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.321+0000 7fa60e0a8640 10 monclient: shutdown 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.321+0000 7fa60e0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.321+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3367392509 >> v1:172.21.15.22:6790/0 conn(0x7fa6080747c0 legacy=0x7fa60813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.322+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3367392509 shutdown_connections 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.322+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3367392509 >> 172.21.15.22:0/3367392509 conn(0x7fa6080b4200 msgr2=0x7fa6080b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.322+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3367392509 shutdown_connections 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.322+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3367392509 wait complete. 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.323+0000 7fa60e0a8640 10 monclient: build_initial_monmap 2024-03-29T11:03:37.305 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.323+0000 7fa60e0a8640 10 monclient: monmap: 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.323+0000 7fa60e0a8640 1 librados: starting msgr at 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.323+0000 7fa60e0a8640 1 librados: starting objecter 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.323+0000 7fa60e0a8640 1 Processor -- start 2024-03-29T11:03:37.306 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.324+0000 7fa60e0a8640 1 -- start start 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.324+0000 7fa60e0a8640 1 librados: setting wanted keys 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.324+0000 7fa60e0a8640 1 librados: calling monclient init 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.324+0000 7fa60e0a8640 10 monclient: init 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.324+0000 7fa60e0a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 10 monclient(hunting): picked mon.noname-b con 0x7fa6080743e0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 10 monclient(hunting): picked mon.noname-c con 0x7fa60813e680 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 10 monclient(hunting): picked mon.noname-a con 0x7fa6080aa160 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080ab0e0 con 0x7fa6080aa160 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080ab3c0 con 0x7fa6080743e0 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6080aa5f0 con 0x7fa60813e680 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:37.307 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.325+0000 7fa60e0a8640 10 monclient(hunting): authenticate will time out at 1318.630200s 2024-03-29T11:03:37.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.328+0000 7fa6077fe640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fa6080743e0 0x7fa6080a98e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:38505/0 (socket says 172.21.15.22:38505) 2024-03-29T11:03:37.308 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.328+0000 7fa6077fe640 1 -- 172.21.15.22:0/3597558563 learned_addr learned my addr 172.21.15.22:0/3597558563 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:37.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 client.?.objecter ms_handle_connect 0x7fa6080743e0 2024-03-29T11:03:37.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:37.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 client.?.objecter ms_handle_connect 0x7fa60813e680 2024-03-29T11:03:37.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:37.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1427697700 0 0) 0x7fa6080aa5f0 con 0x7fa60813e680 2024-03-29T11:03:37.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:37.311 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f4004eb0 con 0x7fa60813e680 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4252910046 0 0) 0x7fa5f4004eb0 con 0x7fa60813e680 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6080aa5f0 con 0x7fa60813e680 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa5fc001b70 con 0x7fa60813e680 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): dump: 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.312 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.329+0000 7fa5effff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.330+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2932118037 0 0) 0x7fa6080aa5f0 con 0x7fa60813e680 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.330+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 >> v1:172.21.15.26:6789/0 conn(0x7fa6080743e0 legacy=0x7fa6080a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:37.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.330+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 >> v1:172.21.15.22:6789/0 conn(0x7fa6080aa160 legacy=0x7fa608140550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:37.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.331+0000 7fa5effff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:37.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.331+0000 7fa5effff640 1 monclient: found mon.c 2024-03-29T11:03:37.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.331+0000 7fa5effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:37.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.331+0000 7fa5effff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:37.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.331+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6081e9e60 con 0x7fa60813e680 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 10 monclient: _finish_auth 0 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 10 monclient: _renew_subs 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6081e6e30 con 0x7fa60813e680 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 10 monclient: _renew_subs 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6081e7310 con 0x7fa60813e680 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa60e0a8640 1 librados: init done 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5fc004bf0 con 0x7fa60813e680 2024-03-29T11:03:37.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa605ffb640 4 set_mon_vals no callback set 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa5fc004f00 con 0x7fa60813e680 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.332+0000 7fa5effff640 10 monclient: dump: 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:37.316 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:37.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.333+0000 7fa60e0a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:37.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.333+0000 7fa60e0a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:37.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.333+0000 7fa60e0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:37.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.333+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5e8005080 con 0x7fa60813e680 2024-03-29T11:03:37.317 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.334+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fa5fc01aed0 con 0x7fa60813e680 2024-03-29T11:03:37.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.335+0000 7fa6077fe640 10 monclient: get_auth_request con 0x7fa5f406dd20 auth_method 0 2024-03-29T11:03:37.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.335+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fa5fc084b30 con 0x7fa60813e680 2024-03-29T11:03:37.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.335+0000 7fa5effff640 10 client.4673.objecter ms_dispatch 0x7fa608000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:37.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.335+0000 7fa5effff640 3 client.4673.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:37.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.335+0000 7fa5effff640 3 client.4673.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:37.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.335+0000 7fa5effff640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:03:37.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.335+0000 7fa5effff640 10 client.4673.objecter ms_handle_connect 0x7fa5f406dd20 2024-03-29T11:03:37.322 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.342+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5fc050490 con 0x7fa60813e680 2024-03-29T11:03:37.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.342+0000 7fa5effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:37.323 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.342+0000 7fa5effff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:37.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.676+0000 7fa60e0a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:37.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.676+0000 7fa60e0a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:37.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.676+0000 7fa60e0a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:37.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.676+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5e8005330 con 0x7fa60813e680 2024-03-29T11:03:37.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.677+0000 7fa5effff640 1 -- 172.21.15.22:0/3597558563 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fa5fc0523c0 con 0x7fa60813e680 2024-03-29T11:03:37.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.677+0000 7fa5effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:37.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.677+0000 7fa5effff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:37.658 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:37.658 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:37.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 10 librados: watch_flush enter 2024-03-29T11:03:37.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 10 librados: watch_flush exit 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 >> v1:172.21.15.22:6816/380221644 conn(0x7fa5f406dd20 legacy=0x7fa5f40701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 10 monclient: shutdown 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 >> v1:172.21.15.22:6790/0 conn(0x7fa60813e680 legacy=0x7fa60813fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 shutdown_connections 2024-03-29T11:03:37.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.681+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 >> 172.21.15.22:0/3597558563 conn(0x7fa6080b4200 msgr2=0x7fa60813ead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:37.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.684+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 shutdown_connections 2024-03-29T11:03:37.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.684+0000 7fa60e0a8640 1 -- 172.21.15.22:0/3597558563 wait complete. 2024-03-29T11:03:37.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:37.684+0000 7fa60e0a8640 1 librados: shutdown 2024-03-29T11:03:37.680 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:44.487 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.652+0000 7f2afe367640 10 monclient: get_monmap_and_config 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.652+0000 7f2afe367640 10 monclient: build_initial_monmap 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.652+0000 7f2afe367640 10 monclient: monmap: 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:44.654463+0000 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:44.654463+0000 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:44.633 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:44.634 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:44.634 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:44.634 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:44.634 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:44.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.653+0000 7f2afe367640 1 Processor -- start 2024-03-29T11:03:44.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.654+0000 7f2afe367640 1 -- start start 2024-03-29T11:03:44.634 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.654+0000 7f2afe367640 10 monclient: init 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 10 monclient(hunting): picked mon.noname-a con 0x7f2af81422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 10 monclient(hunting): picked mon.noname-c con 0x7f2af8142690 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 10 monclient(hunting): picked mon.noname-b con 0x7f2af80b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af8142c80 con 0x7f2af81422b0 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af80aae40 con 0x7f2af8142690 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af813fbd0 con 0x7f2af80b17d0 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:44.648 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2afe367640 10 monclient(hunting): authenticate will time out at 1325.960220s 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af7fff640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f2af81422b0 0x7f2af813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:49224/0 (socket says 172.21.15.22:49224) 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af7fff640 1 -- 172.21.15.22:0/3646453611 learned_addr learned my addr 172.21.15.22:0/3646453611 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4021447308 0 0) 0x7f2af8142c80 con 0x7f2af81422b0 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae0003520 con 0x7f2af81422b0 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542278085 0 0) 0x7f2af80aae40 con 0x7f2af8142690 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.655+0000 7f2af6ffd640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af8142c80 con 0x7f2af8142690 2024-03-29T11:03:44.649 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202203548 0 0) 0x7f2af813fbd0 con 0x7f2af80b17d0 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af80aae40 con 0x7f2af80b17d0 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948097209 0 0) 0x7f2ae0003520 con 0x7f2af81422b0 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 >> v1:172.21.15.22:6790/0 conn(0x7f2af80b17d0 legacy=0x7f2af80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 >> v1:172.21.15.26:6789/0 conn(0x7f2af8142690 legacy=0x7f2af80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 monclient: found mon.noname-a 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:03:44.650 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af8151940 con 0x7f2af81422b0 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2ae8001690 con 0x7f2af81422b0 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: dump: 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:44.651 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2afe367640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2afe367640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ae80017f0 con 0x7f2af81422b0 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 1 -- 172.21.15.22:0/3646453611 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2ae8001b80 con 0x7f2af81422b0 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.656+0000 7f2af6ffd640 10 monclient: dump: 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.652 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.657+0000 7f2afe367640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.657+0000 7f2afe367640 4 set_mon_vals no callback set 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.657+0000 7f2afe367640 10 monclient: shutdown 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.657+0000 7f2afe367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.657+0000 7f2afe367640 1 -- 172.21.15.22:0/3646453611 >> v1:172.21.15.22:6789/0 conn(0x7f2af81422b0 legacy=0x7f2af813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.658+0000 7f2afe367640 1 -- 172.21.15.22:0/3646453611 shutdown_connections 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.658+0000 7f2afe367640 1 -- 172.21.15.22:0/3646453611 >> 172.21.15.22:0/3646453611 conn(0x7f2af8066a00 msgr2=0x7f2af8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.658+0000 7f2afe367640 1 -- 172.21.15.22:0/3646453611 shutdown_connections 2024-03-29T11:03:44.653 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.658+0000 7f2afe367640 1 -- 172.21.15.22:0/3646453611 wait complete. 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 10 monclient: build_initial_monmap 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 10 monclient: monmap: 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 1 librados: starting msgr at 2024-03-29T11:03:44.654 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 1 librados: starting objecter 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 1 Processor -- start 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 1 -- start start 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 1 librados: setting wanted keys 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 1 librados: calling monclient init 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.659+0000 7f2afe367640 10 monclient: init 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 10 monclient(hunting): picked mon.noname-b con 0x7f2af80b17d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 10 monclient(hunting): picked mon.noname-a con 0x7f2af81422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 10 monclient(hunting): picked mon.noname-c con 0x7f2af8142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af80b5b20 con 0x7f2af81422b0 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af806e030 con 0x7f2af80b17d0 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af80b66c0 con 0x7f2af8142690 2024-03-29T11:03:44.655 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.660+0000 7f2afe367640 10 monclient(hunting): authenticate will time out at 1325.966037s 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af77fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f2af81422b0 0x7f2af81e8020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:49752/0 (socket says 172.21.15.22:49752) 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af77fe640 1 -- 172.21.15.22:0/734614202 learned_addr learned my addr 172.21.15.22:0/734614202 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 client.?.objecter ms_handle_connect 0x7f2af81422b0 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 client.?.objecter ms_handle_connect 0x7f2af8142690 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 client.?.objecter ms_handle_connect 0x7f2af80b17d0 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819231984 0 0) 0x7f2af80b66c0 con 0x7f2af8142690 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:44.656 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae4004eb0 con 0x7f2af8142690 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.661+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3038092112 0 0) 0x7f2af806e030 con 0x7f2af80b17d0 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae40051d0 con 0x7f2af80b17d0 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1284035466 0 0) 0x7f2af80b5b20 con 0x7f2af81422b0 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): my global_id is 4512 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2af806e030 con 0x7f2af81422b0 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2582393342 0 0) 0x7f2ae40051d0 con 0x7f2af80b17d0 2024-03-29T11:03:44.657 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2af80b5b20 con 0x7f2af80b17d0 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2ae8001b00 con 0x7f2af80b17d0 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): dump: 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:44.658 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2536807813 0 0) 0x7f2ae4004eb0 con 0x7f2af8142690 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae40051d0 con 0x7f2af8142690 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2af0001c70 con 0x7f2af8142690 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.662+0000 7f2af4ff9640 10 monclient(hunting): dump: 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:44.659 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1426579378 0 0) 0x7f2ae40051d0 con 0x7f2af8142690 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 >> v1:172.21.15.26:6789/0 conn(0x7f2af80b17d0 legacy=0x7f2af81e7910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 >> v1:172.21.15.22:6789/0 conn(0x7f2af81422b0 legacy=0x7f2af81e8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 1 monclient: found mon.c 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af80b6850 con 0x7f2af8142690 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2af4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 5 monclient: authenticate success, global_id 4679 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 10 monclient: _renew_subs 2024-03-29T11:03:44.660 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af80b6b20 con 0x7f2af8142690 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 10 client.4679.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 10 monclient: _renew_subs 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af81efdd0 con 0x7f2af8142690 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.663+0000 7f2afe367640 1 librados: init done 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.664+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2af0003970 con 0x7f2af8142690 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.664+0000 7f2af4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.664+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2af0001c70 con 0x7f2af8142690 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.664+0000 7f2af4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.664+0000 7f2af4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.664+0000 7f2af5ffb640 4 set_mon_vals no callback set 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.664+0000 7f2af4ff9640 10 monclient: dump: 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:44.661 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.665+0000 7f2afe367640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.665+0000 7f2afe367640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.665+0000 7f2afe367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.665+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad8005080 con 0x7f2af8142690 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.666+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f2af0019950 con 0x7f2af8142690 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.666+0000 7f2af7fff640 10 monclient: get_auth_request con 0x7f2ae406e2a0 auth_method 0 2024-03-29T11:03:44.662 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.667+0000 7f2af4ff9640 10 client.4679.objecter ms_handle_connect 0x7f2ae406e2a0 2024-03-29T11:03:44.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.667+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f2af0001a90 con 0x7f2af8142690 2024-03-29T11:03:44.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.667+0000 7f2af4ff9640 10 client.4679.objecter ms_dispatch 0x7f2af8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:44.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.667+0000 7f2af4ff9640 3 client.4679.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:44.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.667+0000 7f2af4ff9640 3 client.4679.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:44.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.667+0000 7f2af4ff9640 20 client.4679.objecter dump_active .. 0 homeless 2024-03-29T11:03:44.663 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.675+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2af004f6a0 con 0x7f2af8142690 2024-03-29T11:03:44.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.675+0000 7f2af4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:44.664 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:44.675+0000 7f2af4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:45.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.058+0000 7f2afe367640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:45.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.058+0000 7f2afe367640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:45.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.059+0000 7f2afe367640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:45.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.059+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ad8005330 con 0x7f2af8142690 2024-03-29T11:03:45.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.061+0000 7f2af4ff9640 1 -- 172.21.15.22:0/734614202 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f2af004f6a0 con 0x7f2af8142690 2024-03-29T11:03:45.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.061+0000 7f2af4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:45.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.061+0000 7f2af4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:45.042 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:45.042 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:45.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 10 librados: watch_flush enter 2024-03-29T11:03:45.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 10 librados: watch_flush exit 2024-03-29T11:03:45.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 20 client.4679.objecter shutdown clearing up homeless session... 2024-03-29T11:03:45.044 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 10 client.4679.objecter successfully canceled tick 2024-03-29T11:03:45.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 >> v1:172.21.15.22:6816/380221644 conn(0x7f2ae406e2a0 legacy=0x7f2ae4070760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:45.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 10 monclient: shutdown 2024-03-29T11:03:45.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:45.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 >> v1:172.21.15.22:6790/0 conn(0x7f2af8142690 legacy=0x7f2af80b5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:45.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 shutdown_connections 2024-03-29T11:03:45.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.064+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 >> 172.21.15.22:0/734614202 conn(0x7f2af8066a00 msgr2=0x7f2af80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:45.045 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.065+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 shutdown_connections 2024-03-29T11:03:45.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.065+0000 7f2afe367640 1 -- 172.21.15.22:0/734614202 wait complete. 2024-03-29T11:03:45.046 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:45.065+0000 7f2afe367640 1 librados: shutdown 2024-03-29T11:03:45.060 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:51.868 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:52.018 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.038+0000 7f59ddd4d640 10 monclient: get_monmap_and_config 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.038+0000 7f59ddd4d640 10 monclient: build_initial_monmap 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.038+0000 7f59ddd4d640 10 monclient: monmap: 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:52.040319+0000 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:52.040319+0000 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:52.019 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:52.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.039+0000 7f59ddd4d640 1 Processor -- start 2024-03-29T11:03:52.020 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.039+0000 7f59ddd4d640 1 -- start start 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.039+0000 7f59ddd4d640 10 monclient: init 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f59d81422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f59d8142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f59d80b17d0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d8142c80 con 0x7f59d81422b0 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80aae40 con 0x7f59d80b17d0 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d813fbd0 con 0x7f59d8142690 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59ddd4d640 10 monclient(hunting): authenticate will time out at 1333.345888s 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59d7fff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f59d80b17d0 0x7f59d80b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:42866/0 (socket says 172.21.15.22:42866) 2024-03-29T11:03:52.021 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.040+0000 7f59d7fff640 1 -- 172.21.15.22:0/587575411 learned_addr learned my addr 172.21.15.22:0/587575411 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.041+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738507749 0 0) 0x7f59d80aae40 con 0x7f59d80b17d0 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.041+0000 7f59d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.041+0000 7f59d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.041+0000 7f59d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.041+0000 7f59d67fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c0003520 con 0x7f59d80b17d0 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614666452 0 0) 0x7f59d8142c80 con 0x7f59d81422b0 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:03:52.025 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d80aae40 con 0x7f59d81422b0 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3362504064 0 0) 0x7f59c0003520 con 0x7f59d80b17d0 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 >> v1:172.21.15.22:6790/0 conn(0x7f59d8142690 legacy=0x7f59d80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 >> v1:172.21.15.22:6789/0 conn(0x7f59d81422b0 legacy=0x7f59d813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 monclient: found mon.noname-c 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8151940 con 0x7f59d80b17d0 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: _finish_auth 0 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f59d0001a10 con 0x7f59d80b17d0 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:52.026 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: dump: 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59ddd4d640 5 monclient: authenticate success, global_id 4492 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59ddd4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59d0001b70 con 0x7f59d80b17d0 2024-03-29T11:03:52.027 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 1 -- 172.21.15.22:0/587575411 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f59d0001f00 con 0x7f59d80b17d0 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59d67fc640 10 monclient: dump: 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:52.028 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59ddd4d640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.042+0000 7f59ddd4d640 4 set_mon_vals no callback set 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.043+0000 7f59ddd4d640 10 monclient: shutdown 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.043+0000 7f59ddd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.043+0000 7f59ddd4d640 1 -- 172.21.15.22:0/587575411 >> v1:172.21.15.26:6789/0 conn(0x7f59d80b17d0 legacy=0x7f59d80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.043+0000 7f59ddd4d640 1 -- 172.21.15.22:0/587575411 shutdown_connections 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.043+0000 7f59ddd4d640 1 -- 172.21.15.22:0/587575411 >> 172.21.15.22:0/587575411 conn(0x7f59d8066a00 msgr2=0x7f59d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.044+0000 7f59ddd4d640 1 -- 172.21.15.22:0/587575411 shutdown_connections 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.044+0000 7f59ddd4d640 1 -- 172.21.15.22:0/587575411 wait complete. 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 10 monclient: build_initial_monmap 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 10 monclient: monmap: 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:52.029 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:52.030 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:52.030 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:52.030 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:52.030 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:52.030 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:52.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 1 librados: starting msgr at 2024-03-29T11:03:52.035 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 1 librados: starting objecter 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 1 Processor -- start 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 1 -- start start 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 1 librados: setting wanted keys 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 1 librados: calling monclient init 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.045+0000 7f59ddd4d640 10 monclient: init 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.046+0000 7f59ddd4d640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.046+0000 7f59ddd4d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.046+0000 7f59ddd4d640 10 monclient(hunting): picked mon.noname-a con 0x7f59d80b17d0 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.046+0000 7f59ddd4d640 10 monclient(hunting): picked mon.noname-b con 0x7f59d81422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59d77fe640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f59d80b17d0 0x7f59d806e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:49156/0 (socket says 172.21.15.22:49156) 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59ddd4d640 10 monclient(hunting): picked mon.noname-c con 0x7f59d8142690 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59ddd4d640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d8072020 con 0x7f59d80b17d0 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59ddd4d640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d8070570 con 0x7f59d81422b0 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59d77fe640 1 -- 172.21.15.22:0/448613930 learned_addr learned my addr 172.21.15.22:0/448613930 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d8070850 con 0x7f59d8142690 2024-03-29T11:03:52.036 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59ddd4d640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59ddd4d640 10 monclient(hunting): authenticate will time out at 1333.352340s 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 client.?.objecter ms_handle_connect 0x7f59d80b17d0 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 client.?.objecter ms_handle_connect 0x7f59d81422b0 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169603771 0 0) 0x7f59d8070570 con 0x7f59d81422b0 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c4004eb0 con 0x7f59d81422b0 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 client.?.objecter ms_handle_connect 0x7f59d8142690 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:52.037 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612350974 0 0) 0x7f59d8072020 con 0x7f59d80b17d0 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): _negotiate 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.047+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59d8070570 con 0x7f59d80b17d0 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3141237700 0 0) 0x7f59c4004eb0 con 0x7f59d81422b0 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59d8072020 con 0x7f59d81422b0 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f59cc001890 con 0x7f59d81422b0 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 10 monclient(hunting): dump: 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:52.038 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655458629 0 0) 0x7f59d8070570 con 0x7f59d80b17d0 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59c4004eb0 con 0x7f59d80b17d0 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3400480386 0 0) 0x7f59d8072020 con 0x7f59d81422b0 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.048+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 >> v1:172.21.15.22:6790/0 conn(0x7f59d8142690 legacy=0x7f59d81e6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 >> v1:172.21.15.22:6789/0 conn(0x7f59d80b17d0 legacy=0x7f59d806e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: _finish_hunting 0 2024-03-29T11:03:52.039 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 1 monclient: found mon.b 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8071c80 con 0x7f59d81422b0 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: _finish_auth 0 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: _check_auth_tickets 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 10 monclient: _renew_subs 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d81df7e0 con 0x7f59d81422b0 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 10 client.4495.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 10 monclient: _renew_subs 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:52.040 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d81defa0 con 0x7f59d81422b0 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59ddd4d640 1 librados: init done 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59cc0035f0 con 0x7f59d81422b0 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59d5ffb640 4 set_mon_vals no callback set 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f59cc001d80 con 0x7f59d81422b0 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.049+0000 7f59bffff640 10 monclient: dump: 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:52.041 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.051+0000 7f59ddd4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.051+0000 7f59ddd4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.051+0000 7f59ddd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.051+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b8005080 con 0x7f59d81422b0 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.051+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f59cc019770 con 0x7f59d81422b0 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.052+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f59cc083690 con 0x7f59d81422b0 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.052+0000 7f59bffff640 10 client.4495.objecter ms_dispatch 0x7f59d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.052+0000 7f59bffff640 3 client.4495.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.052+0000 7f59bffff640 3 client.4495.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.052+0000 7f59bffff640 20 client.4495.objecter dump_active .. 0 homeless 2024-03-29T11:03:52.042 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.052+0000 7f59d77fe640 10 monclient: get_auth_request con 0x7f59c406ddf0 auth_method 0 2024-03-29T11:03:52.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.053+0000 7f59bffff640 10 client.4495.objecter ms_handle_connect 0x7f59c406ddf0 2024-03-29T11:03:52.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.060+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59cc04efd0 con 0x7f59d81422b0 2024-03-29T11:03:52.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.060+0000 7f59bffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:52.043 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.060+0000 7f59bffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:52.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.482+0000 7f59ddd4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:52.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.482+0000 7f59ddd4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:52.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.482+0000 7f59ddd4d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:03:52.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.482+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59b8005330 con 0x7f59d81422b0 2024-03-29T11:03:52.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.483+0000 7f59bffff640 1 -- 172.21.15.22:0/448613930 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f59cc050f00 con 0x7f59d81422b0 2024-03-29T11:03:52.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.483+0000 7f59bffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:52.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.483+0000 7f59bffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:52.466 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:52.466 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:52.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 10 librados: watch_flush enter 2024-03-29T11:03:52.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 10 librados: watch_flush exit 2024-03-29T11:03:52.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 20 client.4495.objecter shutdown clearing up homeless session... 2024-03-29T11:03:52.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 10 client.4495.objecter successfully canceled tick 2024-03-29T11:03:52.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 >> v1:172.21.15.22:6816/380221644 conn(0x7f59c406ddf0 legacy=0x7f59c40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:52.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 10 monclient: shutdown 2024-03-29T11:03:52.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:52.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 >> v1:172.21.15.26:6789/0 conn(0x7f59d81422b0 legacy=0x7f59d806ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:52.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 shutdown_connections 2024-03-29T11:03:52.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 >> 172.21.15.22:0/448613930 conn(0x7f59d8066a00 msgr2=0x7f59d8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:52.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.487+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 shutdown_connections 2024-03-29T11:03:52.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.488+0000 7f59ddd4d640 1 -- 172.21.15.22:0/448613930 wait complete. 2024-03-29T11:03:52.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:52.488+0000 7f59ddd4d640 1 librados: shutdown 2024-03-29T11:03:52.485 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:03:59.293 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.474+0000 7f28a579c640 10 monclient: get_monmap_and_config 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.474+0000 7f28a579c640 10 monclient: build_initial_monmap 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.474+0000 7f28a579c640 10 monclient: monmap: 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:03:59.475529+0000 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:03:59.475529+0000 2024-03-29T11:03:59.454 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:59.455 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:59.455 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:59.455 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:03:59.455 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:03:59.455 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:59.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.475+0000 7f28a579c640 1 Processor -- start 2024-03-29T11:03:59.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.475+0000 7f28a579c640 1 -- start start 2024-03-29T11:03:59.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.475+0000 7f28a579c640 10 monclient: init 2024-03-29T11:03:59.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.476+0000 7f28a579c640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:59.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.476+0000 7f28a579c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:03:59.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.477+0000 7f28a579c640 10 monclient(hunting): picked mon.noname-b con 0x7f28a00b0860 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:59.456 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.477+0000 7f28a579c640 10 monclient(hunting): picked mon.noname-a con 0x7f28a013ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:59.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.477+0000 7f289effd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f28a00b0860 0x7f28a013e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:34240/0 (socket says 172.21.15.22:34240) 2024-03-29T11:03:59.459 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.477+0000 7f289effd640 1 -- 172.21.15.22:0/3982941337 learned_addr learned my addr 172.21.15.22:0/3982941337 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f28a579c640 10 monclient(hunting): picked mon.noname-c con 0x7f28a00af8e0 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a00b0c40 con 0x7f28a013ef30 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a00aed10 con 0x7f28a00af8e0 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a00afd50 con 0x7f28a00b0860 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f28a579c640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f28a579c640 10 monclient(hunting): authenticate will time out at 1340.786444s 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3428710158 0 0) 0x7f28a00aed10 con 0x7f28a00af8e0 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f289dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f289dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f289dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f289dffb640 10 monclient(hunting): my global_id is 4498 2024-03-29T11:03:59.462 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.482+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2888003520 con 0x7f28a00af8e0 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112291873 0 0) 0x7f2888003520 con 0x7f28a00af8e0 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 >> v1:172.21.15.22:6790/0 conn(0x7f28a00b0860 legacy=0x7f28a013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 >> v1:172.21.15.22:6789/0 conn(0x7f28a013ef30 legacy=0x7f28a00af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 1 monclient: found mon.noname-c 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a00ab1b0 con 0x7f28a00af8e0 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 10 monclient: _finish_auth 0 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:59.465 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2898001a10 con 0x7f28a00af8e0 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.483+0000 7f289dffb640 10 monclient: dump: 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:59.466 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.484+0000 7f28a579c640 5 monclient: authenticate success, global_id 4498 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.484+0000 7f28a579c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2898001b70 con 0x7f28a00af8e0 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f289dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f289dffb640 1 -- 172.21.15.22:0/3982941337 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2898001fc0 con 0x7f28a00af8e0 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f289dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f289dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f289dffb640 10 monclient: dump: 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:59.467 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:59.468 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:59.468 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:59.468 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f28a579c640 10 monclient: get_monmap_and_config success 2024-03-29T11:03:59.468 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.485+0000 7f28a579c640 4 set_mon_vals no callback set 2024-03-29T11:03:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.486+0000 7f28a579c640 10 monclient: shutdown 2024-03-29T11:03:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.486+0000 7f28a579c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.486+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 >> v1:172.21.15.26:6789/0 conn(0x7f28a00af8e0 legacy=0x7f28a0149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.486+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 shutdown_connections 2024-03-29T11:03:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.486+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 >> 172.21.15.22:0/3982941337 conn(0x7f28a006b660 msgr2=0x7f28a006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:59.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.488+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 shutdown_connections 2024-03-29T11:03:59.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.490+0000 7f28a579c640 1 -- 172.21.15.22:0/3982941337 wait complete. 2024-03-29T11:03:59.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.491+0000 7f28a579c640 10 monclient: build_initial_monmap 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.491+0000 7f28a579c640 10 monclient: monmap: 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.492+0000 7f28a579c640 1 librados: starting msgr at 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.492+0000 7f28a579c640 1 librados: starting objecter 2024-03-29T11:03:59.472 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.493+0000 7f28a579c640 1 Processor -- start 2024-03-29T11:03:59.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.493+0000 7f28a579c640 1 -- start start 2024-03-29T11:03:59.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.493+0000 7f28a579c640 1 librados: setting wanted keys 2024-03-29T11:03:59.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.493+0000 7f28a579c640 1 librados: calling monclient init 2024-03-29T11:03:59.473 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.493+0000 7f28a579c640 10 monclient: init 2024-03-29T11:03:59.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.494+0000 7f28a579c640 10 monclient: _reopen_session rank -1 2024-03-29T11:03:59.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.494+0000 7f28a579c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:03:59.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.494+0000 7f28a579c640 10 monclient(hunting): picked mon.noname-c con 0x7f28a00af8e0 addr v1:172.21.15.22:6790/0 2024-03-29T11:03:59.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.494+0000 7f28a579c640 10 monclient(hunting): picked mon.noname-b con 0x7f28a00b0860 addr v1:172.21.15.26:6789/0 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f289effd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f28a00af8e0 0x7f28a013a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35394/0 (socket says 172.21.15.22:35394) 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f28a579c640 10 monclient(hunting): picked mon.noname-a con 0x7f28a013ef30 addr v1:172.21.15.22:6789/0 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f289effd640 1 -- 172.21.15.22:0/2492235421 learned_addr learned my addr 172.21.15.22:0/2492235421 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a01d6c80 con 0x7f28a013ef30 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a00b4230 con 0x7f28a00b0860 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a01357d0 con 0x7f28a00af8e0 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f28a579c640 10 monclient(hunting): _renew_subs 2024-03-29T11:03:59.475 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.495+0000 7f28a579c640 10 monclient(hunting): authenticate will time out at 1340.799909s 2024-03-29T11:03:59.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 client.?.objecter ms_handle_connect 0x7f28a00af8e0 2024-03-29T11:03:59.477 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314588914 0 0) 0x7f28a01357d0 con 0x7f28a00af8e0 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f288c004eb0 con 0x7f28a00af8e0 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 client.?.objecter ms_handle_connect 0x7f28a013ef30 2024-03-29T11:03:59.478 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2637633603 0 0) 0x7f288c004eb0 con 0x7f28a00af8e0 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.498+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28a01357d0 con 0x7f28a00af8e0 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2890001950 con 0x7f28a00af8e0 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): dump: 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.479 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982637619 0 0) 0x7f28a01d6c80 con 0x7f28a013ef30 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): _negotiate 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f288c0056b0 con 0x7f28a013ef30 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896507684 0 0) 0x7f288c0056b0 con 0x7f28a013ef30 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28a01d6c80 con 0x7f28a013ef30 2024-03-29T11:03:59.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2898001fc0 con 0x7f28a013ef30 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 10 monclient(hunting): dump: 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1877114703 0 0) 0x7f28a01357d0 con 0x7f28a00af8e0 2024-03-29T11:03:59.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.499+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 >> v1:172.21.15.26:6789/0 conn(0x7f28a00b0860 legacy=0x7f28a00b2190 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:03:59.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 >> v1:172.21.15.22:6789/0 conn(0x7f28a013ef30 legacy=0x7f28a00b28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:59.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 10 monclient: _finish_hunting 0 2024-03-29T11:03:59.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 1 monclient: found mon.c 2024-03-29T11:03:59.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:03:59.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:59.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a01d7110 con 0x7f28a00af8e0 2024-03-29T11:03:59.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 10 monclient: _finish_auth 0 2024-03-29T11:03:59.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 10 monclient: _check_auth_tickets 2024-03-29T11:03:59.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.502+0000 7f28877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:03:59.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.503+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28900047e0 con 0x7f28a00af8e0 2024-03-29T11:03:59.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.503+0000 7f28877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:03:59.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.503+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2890001950 con 0x7f28a00af8e0 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.503+0000 7f28877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.503+0000 7f28877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.503+0000 7f28877fe640 10 monclient: dump: 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:03:59.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.504+0000 7f289d7fa640 4 set_mon_vals no callback set 2024-03-29T11:03:59.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.505+0000 7f28a579c640 5 monclient: authenticate success, global_id 4691 2024-03-29T11:03:59.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.505+0000 7f28a579c640 10 monclient: _renew_subs 2024-03-29T11:03:59.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.505+0000 7f28a579c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:59.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.505+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28a013ac90 con 0x7f28a00af8e0 2024-03-29T11:03:59.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.506+0000 7f28a579c640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:03:59.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.506+0000 7f28a579c640 10 monclient: _renew_subs 2024-03-29T11:03:59.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.506+0000 7f28a579c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:59.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.506+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a01d6560 con 0x7f28a00af8e0 2024-03-29T11:03:59.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.506+0000 7f28a579c640 1 librados: init done 2024-03-29T11:03:59.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.509+0000 7f28a579c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:59.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.509+0000 7f28a579c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:59.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.509+0000 7f28a579c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:59.489 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.509+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a0075f10 con 0x7f28a00af8e0 2024-03-29T11:03:59.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.509+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f289001a840 con 0x7f28a00af8e0 2024-03-29T11:03:59.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.510+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f2890084880 con 0x7f28a00af8e0 2024-03-29T11:03:59.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.510+0000 7f28877fe640 10 client.4691.objecter ms_dispatch 0x7f28a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:03:59.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.510+0000 7f28877fe640 3 client.4691.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:03:59.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.510+0000 7f28877fe640 3 client.4691.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:03:59.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.510+0000 7f28877fe640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-29T11:03:59.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.513+0000 7f289e7fc640 10 monclient: get_auth_request con 0x7f288c0728e0 auth_method 0 2024-03-29T11:03:59.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.514+0000 7f28877fe640 10 client.4691.objecter ms_handle_connect 0x7f288c0728e0 2024-03-29T11:03:59.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.519+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f289004ff90 con 0x7f28a00af8e0 2024-03-29T11:03:59.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.519+0000 7f28877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:03:59.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.519+0000 7f28877fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:03:59.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.870+0000 7f28a579c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:59.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.870+0000 7f28a579c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:59.849 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.870+0000 7f28a579c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:03:59.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.870+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28a0072330 con 0x7f28a00af8e0 2024-03-29T11:03:59.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.872+0000 7f28877fe640 1 -- 172.21.15.22:0/2492235421 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f289004ff90 con 0x7f28a00af8e0 2024-03-29T11:03:59.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.872+0000 7f28877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:03:59.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.872+0000 7f28877fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:03:59.852 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:03:59.852 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:03:59.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 10 librados: watch_flush enter 2024-03-29T11:03:59.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 10 librados: watch_flush exit 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 10 client.4691.objecter successfully canceled tick 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 >> v1:172.21.15.22:6816/380221644 conn(0x7f288c0728e0 legacy=0x7f288c074da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 10 monclient: shutdown 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 >> v1:172.21.15.22:6790/0 conn(0x7f28a00af8e0 legacy=0x7f28a013a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 shutdown_connections 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 >> 172.21.15.22:0/2492235421 conn(0x7f28a006b660 msgr2=0x7f28a01c91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 shutdown_connections 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 1 -- 172.21.15.22:0/2492235421 wait complete. 2024-03-29T11:03:59.856 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:03:59.876+0000 7f28a579c640 1 librados: shutdown 2024-03-29T11:03:59.873 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:06.680 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:06.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.861+0000 7f53eee2e640 10 monclient: get_monmap_and_config 2024-03-29T11:04:06.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.861+0000 7f53eee2e640 10 monclient: build_initial_monmap 2024-03-29T11:04:06.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.861+0000 7f53eee2e640 10 monclient: monmap: 2024-03-29T11:04:06.842 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:06.862568+0000 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:06.862568+0000 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:06.843 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:06.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.863+0000 7f53eee2e640 1 Processor -- start 2024-03-29T11:04:06.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.863+0000 7f53eee2e640 1 -- start start 2024-03-29T11:04:06.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.863+0000 7f53eee2e640 10 monclient: init 2024-03-29T11:04:06.844 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:06.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:06.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 10 monclient(hunting): picked mon.noname-c con 0x7f53e80a9930 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:06.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 10 monclient(hunting): picked mon.noname-a con 0x7f53e8149dc0 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:06.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 10 monclient(hunting): picked mon.noname-b con 0x7f53e814ff40 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:06.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e80aae60 con 0x7f53e8149dc0 2024-03-29T11:04:06.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e80b4310 con 0x7f53e80a9930 2024-03-29T11:04:06.845 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e80afdd0 con 0x7f53e814ff40 2024-03-29T11:04:06.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:06.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.864+0000 7f53eee2e640 10 monclient(hunting): authenticate will time out at 1348.168543s 2024-03-29T11:04:06.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.866+0000 7f53ed3cb640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f53e814ff40 0x7f53e8152330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57988/0 (socket says 172.21.15.22:57988) 2024-03-29T11:04:06.846 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.866+0000 7f53ed3cb640 1 -- 172.21.15.22:0/935576973 learned_addr learned my addr 172.21.15.22:0/935576973 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:06.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.868+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162743821 0 0) 0x7f53e80afdd0 con 0x7f53e814ff40 2024-03-29T11:04:06.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.868+0000 7f53e77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:06.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.868+0000 7f53e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:06.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.868+0000 7f53e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:06.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.868+0000 7f53e77fe640 10 monclient(hunting): my global_id is 4694 2024-03-29T11:04:06.847 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.868+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d0003520 con 0x7f53e814ff40 2024-03-29T11:04:06.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2008304677 0 0) 0x7f53d0003520 con 0x7f53e814ff40 2024-03-29T11:04:06.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 >> v1:172.21.15.26:6789/0 conn(0x7f53e80a9930 legacy=0x7f53e8143a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:06.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 >> v1:172.21.15.22:6789/0 conn(0x7f53e8149dc0 legacy=0x7f53e814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:06.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:06.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 1 monclient: found mon.noname-b 2024-03-29T11:04:06.850 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e814a950 con 0x7f53e814ff40 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53eee2e640 5 monclient: authenticate success, global_id 4694 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53eee2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53e0001690 con 0x7f53e814ff40 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.869+0000 7f53e77fe640 10 monclient: dump: 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:06.851 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.870+0000 7f53eee2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.871+0000 7f53e77fe640 1 -- 172.21.15.22:0/935576973 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53e0001860 con 0x7f53e814ff40 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.871+0000 7f53e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:06.852 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.871+0000 7f53eee2e640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:06.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.871+0000 7f53eee2e640 4 set_mon_vals no callback set 2024-03-29T11:04:06.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.872+0000 7f53eee2e640 10 monclient: shutdown 2024-03-29T11:04:06.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.872+0000 7f53eee2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:06.853 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.872+0000 7f53eee2e640 1 -- 172.21.15.22:0/935576973 >> v1:172.21.15.22:6790/0 conn(0x7f53e814ff40 legacy=0x7f53e8152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:06.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.873+0000 7f53eee2e640 1 -- 172.21.15.22:0/935576973 shutdown_connections 2024-03-29T11:04:06.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.873+0000 7f53eee2e640 1 -- 172.21.15.22:0/935576973 >> 172.21.15.22:0/935576973 conn(0x7f53e80b35e0 msgr2=0x7f53e80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:06.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.874+0000 7f53eee2e640 1 -- 172.21.15.22:0/935576973 shutdown_connections 2024-03-29T11:04:06.855 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.874+0000 7f53eee2e640 1 -- 172.21.15.22:0/935576973 wait complete. 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.875+0000 7f53eee2e640 10 monclient: build_initial_monmap 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.875+0000 7f53eee2e640 10 monclient: monmap: 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:06.858 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.876+0000 7f53eee2e640 1 librados: starting msgr at 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.876+0000 7f53eee2e640 1 librados: starting objecter 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.876+0000 7f53eee2e640 1 Processor -- start 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.876+0000 7f53eee2e640 1 -- start start 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.876+0000 7f53eee2e640 1 librados: setting wanted keys 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.876+0000 7f53eee2e640 1 librados: calling monclient init 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.876+0000 7f53eee2e640 10 monclient: init 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:06.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 10 monclient(hunting): picked mon.noname-c con 0x7f53e80a9930 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 10 monclient(hunting): picked mon.noname-a con 0x7f53e8149dc0 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 10 monclient(hunting): picked mon.noname-b con 0x7f53e814ff40 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e8147180 con 0x7f53e8149dc0 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e8145130 con 0x7f53e814ff40 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53e813dca0 con 0x7f53e80a9930 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53eee2e640 10 monclient(hunting): authenticate will time out at 1348.181485s 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53ecbca640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f53e80a9930 0x7f53e8142b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59884/0 (socket says 172.21.15.22:59884) 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.877+0000 7f53ecbca640 1 -- 172.21.15.22:0/1215187305 learned_addr learned my addr 172.21.15.22:0/1215187305 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 10 client.?.objecter ms_handle_connect 0x7f53e80a9930 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211962349 0 0) 0x7f53e813dca0 con 0x7f53e80a9930 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:04:06.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 10 monclient(hunting): my global_id is 4697 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.878+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53d4004eb0 con 0x7f53e80a9930 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 178779340 0 0) 0x7f53d4004eb0 con 0x7f53e80a9930 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53e813dca0 con 0x7f53e80a9930 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53d8001bf0 con 0x7f53e80a9930 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 10 monclient(hunting): dump: 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:06.861 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1701324446 0 0) 0x7f53e813dca0 con 0x7f53e80a9930 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.879+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 >> v1:172.21.15.26:6789/0 conn(0x7f53e814ff40 legacy=0x7f53e82000a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:04:06.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 >> v1:172.21.15.22:6789/0 conn(0x7f53e8149dc0 legacy=0x7f53e813d590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:04:06.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 1 monclient: found mon.c 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e8144d90 con 0x7f53e80a9930 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 10 monclient: _finish_auth 0 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.882+0000 7f53e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 5 monclient: authenticate success, global_id 4697 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 10 monclient: _renew_subs 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53e813cc10 con 0x7f53e80a9930 2024-03-29T11:04:06.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 10 monclient: _renew_subs 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53e8144510 con 0x7f53e80a9930 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.883+0000 7f53eee2e640 1 librados: init done 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53d8001d50 con 0x7f53e80a9930 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53eee2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f53d8004e60 con 0x7f53e80a9930 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53eee2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:06.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53eee2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53e6ffd640 4 set_mon_vals no callback set 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.884+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53e8075f10 con 0x7f53e80a9930 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.885+0000 7f53e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.885+0000 7f53e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.885+0000 7f53e57fa640 10 monclient: dump: 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:06.866 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:06.867 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:06.867 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:06.867 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:06.867 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:06.867 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:06.867 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:06.867 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.888+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f53d8004fc0 con 0x7f53e80a9930 2024-03-29T11:04:06.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.888+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f53d8084e30 con 0x7f53e80a9930 2024-03-29T11:04:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.888+0000 7f53e57fa640 10 client.4697.objecter ms_dispatch 0x7f53e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.888+0000 7f53e57fa640 3 client.4697.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.888+0000 7f53e57fa640 3 client.4697.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.888+0000 7f53e57fa640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-29T11:04:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.890+0000 7f53e7fff640 10 monclient: get_auth_request con 0x7f53d406dda0 auth_method 0 2024-03-29T11:04:06.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.890+0000 7f53e57fa640 10 client.4697.objecter ms_handle_connect 0x7f53d406dda0 2024-03-29T11:04:06.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.897+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53d8050540 con 0x7f53e80a9930 2024-03-29T11:04:06.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.897+0000 7f53e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:06.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:06.897+0000 7f53e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:07.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.253+0000 7f53eee2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:07.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.253+0000 7f53eee2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:07.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.253+0000 7f53eee2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:07.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.253+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53e813cda0 con 0x7f53e80a9930 2024-03-29T11:04:07.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.258+0000 7f53e57fa640 1 -- 172.21.15.22:0/1215187305 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f53d8052640 con 0x7f53e80a9930 2024-03-29T11:04:07.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.258+0000 7f53e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:07.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.258+0000 7f53e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:07.237 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:07.238 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 10 librados: watch_flush enter 2024-03-29T11:04:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 10 librados: watch_flush exit 2024-03-29T11:04:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-29T11:04:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 10 client.4697.objecter successfully canceled tick 2024-03-29T11:04:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 >> v1:172.21.15.22:6816/380221644 conn(0x7f53d406dda0 legacy=0x7f53d4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 10 monclient: shutdown 2024-03-29T11:04:07.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:07.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.263+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 >> v1:172.21.15.22:6790/0 conn(0x7f53e80a9930 legacy=0x7f53e8142b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:07.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.264+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 shutdown_connections 2024-03-29T11:04:07.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.264+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 >> 172.21.15.22:0/1215187305 conn(0x7f53e80b35e0 msgr2=0x7f53e80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:07.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.265+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 shutdown_connections 2024-03-29T11:04:07.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.265+0000 7f53eee2e640 1 -- 172.21.15.22:0/1215187305 wait complete. 2024-03-29T11:04:07.245 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:07.266+0000 7f53eee2e640 1 librados: shutdown 2024-03-29T11:04:07.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:14.071 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.229+0000 7f752366e640 10 monclient: get_monmap_and_config 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.229+0000 7f752366e640 10 monclient: build_initial_monmap 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.229+0000 7f752366e640 10 monclient: monmap: 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:14.230275+0000 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:14.230275+0000 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:14.209 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:14.210 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:14.210 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:14.210 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:14.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.230+0000 7f752366e640 1 Processor -- start 2024-03-29T11:04:14.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.230+0000 7f752366e640 1 -- start start 2024-03-29T11:04:14.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.230+0000 7f752366e640 10 monclient: init 2024-03-29T11:04:14.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.231+0000 7f752366e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:14.211 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.231+0000 7f752366e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.231+0000 7f752366e640 10 monclient(hunting): picked mon.noname-a con 0x7f751c0ae250 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.231+0000 7f752366e640 10 monclient(hunting): picked mon.noname-c con 0x7f751c0a9720 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.231+0000 7f752366e640 10 monclient(hunting): picked mon.noname-b con 0x7f751c14bba0 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f752140a640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f751c0ae250 0x7f751c142b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38432/0 (socket says 172.21.15.22:38432) 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.231+0000 7f752366e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c0a9b80 con 0x7f751c0ae250 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f752140a640 1 -- 172.21.15.22:0/1990425763 learned_addr learned my addr 172.21.15.22:0/1990425763 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f752366e640 1 -- 172.21.15.22:0/1990425763 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c0aa7b0 con 0x7f751c0a9720 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f752366e640 1 -- 172.21.15.22:0/1990425763 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c0afbf0 con 0x7f751c14bba0 2024-03-29T11:04:14.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f752366e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f752366e640 10 monclient(hunting): authenticate will time out at 1355.536172s 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947544057 0 0) 0x7f751c0afbf0 con 0x7f751c14bba0 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 10 monclient(hunting): my global_id is 4700 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7508003520 con 0x7f751c14bba0 2024-03-29T11:04:14.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2884516337 0 0) 0x7f7508003520 con 0x7f751c14bba0 2024-03-29T11:04:14.214 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.232+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 >> v1:172.21.15.26:6789/0 conn(0x7f751c0a9720 legacy=0x7f751c13f7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 >> v1:172.21.15.22:6789/0 conn(0x7f751c0ae250 legacy=0x7f751c142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 1 monclient: found mon.noname-b 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c143280 con 0x7f751c14bba0 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 10 monclient: _finish_auth 0 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:14.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7518001690 con 0x7f751c14bba0 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.235+0000 7f7513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 10 monclient: dump: 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:14.216 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f752366e640 5 monclient: authenticate success, global_id 4700 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f752366e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75180017f0 con 0x7f751c14bba0 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 1 -- 172.21.15.22:0/1990425763 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f7518001c40 con 0x7f751c14bba0 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f7513fff640 10 monclient: dump: 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.217 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f752366e640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:14.218 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.236+0000 7f752366e640 4 set_mon_vals no callback set 2024-03-29T11:04:14.219 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.237+0000 7f752366e640 10 monclient: shutdown 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.237+0000 7f752366e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.237+0000 7f752366e640 1 -- 172.21.15.22:0/1990425763 >> v1:172.21.15.22:6790/0 conn(0x7f751c14bba0 legacy=0x7f751c14df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.237+0000 7f752366e640 1 -- 172.21.15.22:0/1990425763 shutdown_connections 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.237+0000 7f752366e640 1 -- 172.21.15.22:0/1990425763 >> 172.21.15.22:0/1990425763 conn(0x7f751c0b35e0 msgr2=0x7f751c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.238+0000 7f752366e640 1 -- 172.21.15.22:0/1990425763 shutdown_connections 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.238+0000 7f752366e640 1 -- 172.21.15.22:0/1990425763 wait complete. 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.239+0000 7f752366e640 10 monclient: build_initial_monmap 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.239+0000 7f752366e640 10 monclient: monmap: 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:14.220 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:14.221 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:14.221 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:14.221 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:14.221 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:14.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.240+0000 7f752366e640 1 librados: starting msgr at 2024-03-29T11:04:14.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.240+0000 7f752366e640 1 librados: starting objecter 2024-03-29T11:04:14.221 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.240+0000 7f752366e640 1 Processor -- start 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.240+0000 7f752366e640 1 -- start start 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.240+0000 7f752366e640 1 librados: setting wanted keys 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.241+0000 7f752366e640 1 librados: calling monclient init 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.241+0000 7f752366e640 10 monclient: init 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 10 monclient(hunting): picked mon.noname-a con 0x7f751c0a9720 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 10 monclient(hunting): picked mon.noname-b con 0x7f751c0ae250 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:14.222 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 10 monclient(hunting): picked mon.noname-c con 0x7f751c14bba0 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:14.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c13ba90 con 0x7f751c0a9720 2024-03-29T11:04:14.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c0acf30 con 0x7f751c0ae250 2024-03-29T11:04:14.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c0b2dd0 con 0x7f751c14bba0 2024-03-29T11:04:14.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:14.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752366e640 10 monclient(hunting): authenticate will time out at 1355.547303s 2024-03-29T11:04:14.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752140a640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f751c0a9720 0x7f751c1320a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:38978/0 (socket says 172.21.15.22:38978) 2024-03-29T11:04:14.223 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f752140a640 1 -- 172.21.15.22:0/1481965327 learned_addr learned my addr 172.21.15.22:0/1481965327 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 client.?.objecter ms_handle_connect 0x7f751c0a9720 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 client.?.objecter ms_handle_connect 0x7f751c14bba0 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1231778326 0 0) 0x7f751c13ba90 con 0x7f751c0a9720 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7500004eb0 con 0x7f751c0a9720 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944354635 0 0) 0x7f751c0b2dd0 con 0x7f751c14bba0 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:14.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 10 monclient(hunting): my global_id is 4703 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.243+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f751c13ba90 con 0x7f751c14bba0 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3967492980 0 0) 0x7f7500004eb0 con 0x7f751c0a9720 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f751c0b2dd0 con 0x7f751c0a9720 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f750c0018f0 con 0x7f751c0a9720 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 10 monclient(hunting): dump: 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.226 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1589486099 0 0) 0x7f751c13ba90 con 0x7f751c14bba0 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7500004eb0 con 0x7f751c14bba0 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1642017461 0 0) 0x7f751c0b2dd0 con 0x7f751c0a9720 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 >> v1:172.21.15.22:6790/0 conn(0x7f751c14bba0 legacy=0x7f751c1e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:14.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.244+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 >> v1:172.21.15.26:6789/0 conn(0x7f751c0ae250 legacy=0x7f751c13bff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 1 monclient: found mon.a 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0ad1a0 con 0x7f751c0a9720 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f7511ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 10 monclient: _renew_subs 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:14.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751c13c700 con 0x7f751c0a9720 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 10 monclient: _renew_subs 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f751c0ac550 con 0x7f751c0a9720 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.248+0000 7f752366e640 1 librados: init done 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f750c001a50 con 0x7f751c0a9720 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f7511ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f75137fe640 4 set_mon_vals no callback set 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f750c004980 con 0x7f751c0a9720 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f7511ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f7511ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f7511ffb640 10 monclient: dump: 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:14.232 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.249+0000 7f752366e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.250+0000 7f752366e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.250+0000 7f752366e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.250+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc005080 con 0x7f751c0a9720 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.251+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f750c01a950 con 0x7f751c0a9720 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.251+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7f750c0849c0 con 0x7f751c0a9720 2024-03-29T11:04:14.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.251+0000 7f7511ffb640 10 client.4530.objecter ms_dispatch 0x7f751c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:14.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.251+0000 7f7511ffb640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:14.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.251+0000 7f7511ffb640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:14.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.251+0000 7f7511ffb640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-29T11:04:14.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.251+0000 7f7520c09640 10 monclient: get_auth_request con 0x7f750006dde0 auth_method 0 2024-03-29T11:04:14.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.252+0000 7f7511ffb640 10 client.4530.objecter ms_handle_connect 0x7f750006dde0 2024-03-29T11:04:14.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.258+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f750c0500d0 con 0x7f751c0a9720 2024-03-29T11:04:14.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.258+0000 7f7511ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:14.237 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.258+0000 7f7511ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:14.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.582+0000 7f752366e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:14.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.582+0000 7f752366e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:14.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.582+0000 7f752366e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:14.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.582+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74fc005330 con 0x7f751c0a9720 2024-03-29T11:04:14.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.586+0000 7f7511ffb640 1 -- 172.21.15.22:0/1481965327 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7f750c0521d0 con 0x7f751c0a9720 2024-03-29T11:04:14.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.586+0000 7f7511ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:14.566 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.586+0000 7f7511ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:14.566 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:14.566 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:14.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 10 librados: watch_flush enter 2024-03-29T11:04:14.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 10 librados: watch_flush exit 2024-03-29T11:04:14.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-29T11:04:14.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 10 client.4530.objecter successfully canceled tick 2024-03-29T11:04:14.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 >> v1:172.21.15.22:6816/380221644 conn(0x7f750006dde0 legacy=0x7f75000702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:14.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 10 monclient: shutdown 2024-03-29T11:04:14.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:14.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.590+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 >> v1:172.21.15.22:6789/0 conn(0x7f751c0a9720 legacy=0x7f751c1320a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:14.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.591+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 shutdown_connections 2024-03-29T11:04:14.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.591+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 >> 172.21.15.22:0/1481965327 conn(0x7f751c0b35e0 msgr2=0x7f751c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:14.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.591+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 shutdown_connections 2024-03-29T11:04:14.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.591+0000 7f752366e640 1 -- 172.21.15.22:0/1481965327 wait complete. 2024-03-29T11:04:14.571 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:14.591+0000 7f752366e640 1 librados: shutdown 2024-03-29T11:04:14.585 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:21.392 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:21.530 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.551+0000 7fd086858640 10 monclient: get_monmap_and_config 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.551+0000 7fd086858640 10 monclient: build_initial_monmap 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.551+0000 7fd086858640 10 monclient: monmap: 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:21.552245+0000 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:21.552245+0000 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:21.531 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:21.532 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.552+0000 7fd086858640 1 Processor -- start 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.552+0000 7fd086858640 1 -- start start 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.552+0000 7fd086858640 10 monclient: init 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.553+0000 7fd086858640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.553+0000 7fd086858640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.553+0000 7fd086858640 10 monclient(hunting): picked mon.noname-c con 0x7fd0801457f0 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.553+0000 7fd086858640 10 monclient(hunting): picked mon.noname-b con 0x7fd08013f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.553+0000 7fd086858640 10 monclient(hunting): picked mon.noname-a con 0x7fd0800a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd086858640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0800aaef0 con 0x7fd0800a9d70 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd086858640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd080145bd0 con 0x7fd0801457f0 2024-03-29T11:04:21.535 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd086858640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd080145eb0 con 0x7fd08013f650 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd086858640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd086858640 10 monclient(hunting): authenticate will time out at 1362.858480s 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.553+0000 7fd07f7fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fd08013f650 0x7fd0800b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52064/0 (socket says 172.21.15.22:52064) 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd07f7fe640 1 -- 172.21.15.22:0/405591055 learned_addr learned my addr 172.21.15.22:0/405591055 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.553+0000 7fd07ffff640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd0801457f0 0x7fd08013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:44448/0 (socket says 172.21.15.22:44448) 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3577632203 0 0) 0x7fd080145eb0 con 0x7fd08013f650 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd07effd640 10 monclient(hunting): _negotiate 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd07effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd07effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd07effd640 10 monclient(hunting): my global_id is 4706 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.554+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd068003520 con 0x7fd08013f650 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225456122 0 0) 0x7fd080145bd0 con 0x7fd0801457f0 2024-03-29T11:04:21.536 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 10 monclient(hunting): _negotiate 2024-03-29T11:04:21.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:21.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:21.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:04:21.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd080145eb0 con 0x7fd0801457f0 2024-03-29T11:04:21.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 371116573 0 0) 0x7fd068003520 con 0x7fd08013f650 2024-03-29T11:04:21.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 >> v1:172.21.15.26:6789/0 conn(0x7fd0801457f0 legacy=0x7fd08013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:21.537 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.555+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 >> v1:172.21.15.22:6789/0 conn(0x7fd0800a9d70 legacy=0x7fd08014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: _finish_hunting 0 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 1 monclient: found mon.noname-b 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd080146340 con 0x7fd08013f650 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: _finish_auth 0 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: _check_auth_tickets 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fd070001690 con 0x7fd08013f650 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: dump: 2024-03-29T11:04:21.539 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd086858640 5 monclient: authenticate success, global_id 4706 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd086858640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd0700017f0 con 0x7fd08013f650 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:21.540 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 1 -- 172.21.15.22:0/405591055 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fd070001c00 con 0x7fd08013f650 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd07effd640 10 monclient: dump: 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd086858640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.557+0000 7fd086858640 4 set_mon_vals no callback set 2024-03-29T11:04:21.541 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.558+0000 7fd086858640 10 monclient: shutdown 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.558+0000 7fd086858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.558+0000 7fd086858640 1 -- 172.21.15.22:0/405591055 >> v1:172.21.15.22:6790/0 conn(0x7fd08013f650 legacy=0x7fd0800b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.558+0000 7fd086858640 1 -- 172.21.15.22:0/405591055 shutdown_connections 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.558+0000 7fd086858640 1 -- 172.21.15.22:0/405591055 >> 172.21.15.22:0/405591055 conn(0x7fd080066a00 msgr2=0x7fd080066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.558+0000 7fd086858640 1 -- 172.21.15.22:0/405591055 shutdown_connections 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.558+0000 7fd086858640 1 -- 172.21.15.22:0/405591055 wait complete. 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.559+0000 7fd086858640 10 monclient: build_initial_monmap 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.559+0000 7fd086858640 10 monclient: monmap: 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:21.542 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:21.543 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:21.543 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:21.543 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:21.543 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:21.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.560+0000 7fd086858640 1 librados: starting msgr at 2024-03-29T11:04:21.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.560+0000 7fd086858640 1 librados: starting objecter 2024-03-29T11:04:21.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.560+0000 7fd086858640 1 Processor -- start 2024-03-29T11:04:21.547 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.561+0000 7fd086858640 1 -- start start 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.561+0000 7fd086858640 1 librados: setting wanted keys 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.561+0000 7fd086858640 1 librados: calling monclient init 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.561+0000 7fd086858640 10 monclient: init 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.561+0000 7fd086858640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.561+0000 7fd086858640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 10 monclient(hunting): picked mon.noname-a con 0x7fd0800a9d70 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 10 monclient(hunting): picked mon.noname-c con 0x7fd08013f650 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 10 monclient(hunting): picked mon.noname-b con 0x7fd0801457f0 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0800ac780 con 0x7fd0800a9d70 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd080137790 con 0x7fd0801457f0 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0800ab970 con 0x7fd08013f650 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.562+0000 7fd086858640 10 monclient(hunting): authenticate will time out at 1362.866756s 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd084df5640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7fd0801457f0 0x7fd080137080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:44696/0 (socket says 172.21.15.22:44696) 2024-03-29T11:04:21.548 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd084df5640 1 -- 172.21.15.22:0/2636253077 learned_addr learned my addr 172.21.15.22:0/2636253077 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 client.?.objecter ms_handle_connect 0x7fd0801457f0 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 client.?.objecter ms_handle_connect 0x7fd08013f650 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546458354 0 0) 0x7fd080137790 con 0x7fd0801457f0 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd06c004eb0 con 0x7fd0801457f0 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3839540985 0 0) 0x7fd06c004eb0 con 0x7fd0801457f0 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd080137790 con 0x7fd0801457f0 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.563+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fd078001c60 con 0x7fd0801457f0 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:21.549 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): dump: 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442861946 0 0) 0x7fd0800ab970 con 0x7fd08013f650 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:04:21.550 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 10 monclient(hunting): my global_id is 4709 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd06c005680 con 0x7fd08013f650 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1465806255 0 0) 0x7fd080137790 con 0x7fd0801457f0 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 >> v1:172.21.15.22:6790/0 conn(0x7fd08013f650 legacy=0x7fd080136970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.564+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 >> v1:172.21.15.22:6789/0 conn(0x7fd0800a9d70 legacy=0x7fd080140140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 1 monclient: found mon.b 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0800ac9f0 con 0x7fd0801457f0 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 10 monclient: _finish_auth 0 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd07cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:04:21.551 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 10 monclient: _renew_subs 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0800ab600 con 0x7fd0801457f0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 10 monclient: _renew_subs 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd080137d10 con 0x7fd0801457f0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.565+0000 7fd086858640 1 librados: init done 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.566+0000 7fd086858640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd086858640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd086858640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd080075f10 con 0x7fd0801457f0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd078003970 con 0x7fd0801457f0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd07cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fd078001c60 con 0x7fd0801457f0 2024-03-29T11:04:21.552 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd07cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd07cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd07cff9640 10 monclient: dump: 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.567+0000 7fd07e7fc640 4 set_mon_vals no callback set 2024-03-29T11:04:21.553 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.568+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fd078003be0 con 0x7fd0801457f0 2024-03-29T11:04:21.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.569+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1537457496 0 0) 0x7fd078084910 con 0x7fd0801457f0 2024-03-29T11:04:21.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.569+0000 7fd07cff9640 10 client.4504.objecter ms_dispatch 0x7fd080000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:04:21.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.569+0000 7fd07cff9640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:04:21.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.569+0000 7fd07cff9640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:04:21.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.569+0000 7fd07ffff640 10 monclient: get_auth_request con 0x7fd06c06e240 auth_method 0 2024-03-29T11:04:21.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.569+0000 7fd07cff9640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-29T11:04:21.554 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.570+0000 7fd07cff9640 10 client.4504.objecter ms_handle_connect 0x7fd06c06e240 2024-03-29T11:04:21.555 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.576+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd078050250 con 0x7fd0801457f0 2024-03-29T11:04:21.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.576+0000 7fd07cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:21.556 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.576+0000 7fd07cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:21.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.895+0000 7fd086858640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:21.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.895+0000 7fd086858640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:21.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.895+0000 7fd086858640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:21.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.895+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0800b36d0 con 0x7fd0801457f0 2024-03-29T11:04:21.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.897+0000 7fd07cff9640 1 -- 172.21.15.22:0/2636253077 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3129884510) 0x7fd078050250 con 0x7fd0801457f0 2024-03-29T11:04:21.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.897+0000 7fd07cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:21.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.897+0000 7fd07cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:21.877 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:21.877 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":10,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T10:54:24.725378+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T10:54:24.725378+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:21.879 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.900+0000 7fd086858640 10 librados: watch_flush enter 2024-03-29T11:04:21.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.901+0000 7fd086858640 10 librados: watch_flush exit 2024-03-29T11:04:21.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.901+0000 7fd086858640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-29T11:04:21.880 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.901+0000 7fd086858640 10 client.4504.objecter successfully canceled tick 2024-03-29T11:04:21.883 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 >> v1:172.21.15.22:6816/380221644 conn(0x7fd06c06e240 legacy=0x7fd06c070700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 10 monclient: shutdown 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 >> v1:172.21.15.26:6789/0 conn(0x7fd0801457f0 legacy=0x7fd080137080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 shutdown_connections 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 >> 172.21.15.22:0/2636253077 conn(0x7fd080066a00 msgr2=0x7fd08013aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 shutdown_connections 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.904+0000 7fd086858640 1 -- 172.21.15.22:0/2636253077 wait complete. 2024-03-29T11:04:21.884 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:21.905+0000 7fd086858640 1 librados: shutdown 2024-03-29T11:04:21.899 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:28.706 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:28.857 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.877+0000 7f5e4a3c3640 10 monclient: get_monmap_and_config 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.877+0000 7f5e4a3c3640 10 monclient: build_initial_monmap 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.877+0000 7f5e4a3c3640 10 monclient: monmap: 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:28.878509+0000 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:28.878509+0000 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:28.858 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.859 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:28.859 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:28.859 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:28.859 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.859 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.878+0000 7f5e4a3c3640 1 Processor -- start 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.879+0000 7f5e4a3c3640 1 -- start start 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.879+0000 7f5e4a3c3640 10 monclient: init 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.879+0000 7f5e4a3c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.879+0000 7f5e4a3c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 10 monclient(hunting): picked mon.noname-a con 0x7f5e3c006800 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 10 monclient(hunting): picked mon.noname-b con 0x7f5e3c006be0 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 10 monclient(hunting): picked mon.noname-c con 0x7f5e3c031b60 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c029370 con 0x7f5e3c006800 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c0295e0 con 0x7f5e3c031b60 2024-03-29T11:04:28.860 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c031360 con 0x7f5e3c006be0 2024-03-29T11:04:28.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:28.861 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.880+0000 7f5e4a3c3640 10 monclient(hunting): authenticate will time out at 1370.184166s 2024-03-29T11:04:28.862 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.883+0000 7f5e48bc0640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f5e3c006be0 0x7f5e3c0c4ad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:55222/0 (socket says 172.21.15.22:55222) 2024-03-29T11:04:28.863 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.883+0000 7f5e48bc0640 1 -- 172.21.15.22:0/584334725 learned_addr learned my addr 172.21.15.22:0/584334725 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:28.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935576955 0 0) 0x7f5e3c031360 con 0x7f5e3c006be0 2024-03-29T11:04:28.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:28.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:28.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:28.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): my global_id is 4712 2024-03-29T11:04:28.864 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e30003520 con 0x7f5e3c006be0 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712968328 0 0) 0x7f5e3c029370 con 0x7f5e3c006800 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e300048a0 con 0x7f5e3c006800 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2004377578 0 0) 0x7f5e30003520 con 0x7f5e3c006be0 2024-03-29T11:04:28.865 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.884+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 >> v1:172.21.15.26:6789/0 conn(0x7f5e3c031b60 legacy=0x7f5e3c0d30c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:28.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 >> v1:172.21.15.22:6789/0 conn(0x7f5e3c006800 legacy=0x7f5e3c0cad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:28.868 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 1 monclient: found mon.noname-b 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c031640 con 0x7f5e3c006be0 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 10 monclient: _finish_auth 0 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:28.869 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5e34001670 con 0x7f5e3c006be0 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.888+0000 7f5e3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 10 monclient: dump: 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.870 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e340017d0 con 0x7f5e3c006be0 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 1 -- 172.21.15.22:0/584334725 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5e34001c20 con 0x7f5e3c006be0 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:28.871 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e3bfff640 10 monclient: dump: 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:28.872 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e4a3c3640 5 monclient: authenticate success, global_id 4712 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e4a3c3640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.889+0000 7f5e4a3c3640 4 set_mon_vals no callback set 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.890+0000 7f5e4a3c3640 10 monclient: shutdown 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.890+0000 7f5e4a3c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.890+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/584334725 >> v1:172.21.15.22:6790/0 conn(0x7f5e3c006be0 legacy=0x7f5e3c0c4ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.890+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/584334725 shutdown_connections 2024-03-29T11:04:28.874 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.890+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/584334725 >> 172.21.15.22:0/584334725 conn(0x7f5e3c0058f0 msgr2=0x7f5e3c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:28.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.891+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/584334725 shutdown_connections 2024-03-29T11:04:28.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.894+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/584334725 wait complete. 2024-03-29T11:04:28.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.895+0000 7f5e4a3c3640 10 monclient: build_initial_monmap 2024-03-29T11:04:28.875 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.895+0000 7f5e4a3c3640 10 monclient: monmap: 2024-03-29T11:04:28.875 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:28.875 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.895+0000 7f5e4a3c3640 1 librados: starting msgr at 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.895+0000 7f5e4a3c3640 1 librados: starting objecter 2024-03-29T11:04:28.876 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.895+0000 7f5e4a3c3640 1 Processor -- start 2024-03-29T11:04:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.896+0000 7f5e4a3c3640 1 -- start start 2024-03-29T11:04:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.896+0000 7f5e4a3c3640 1 librados: setting wanted keys 2024-03-29T11:04:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.896+0000 7f5e4a3c3640 1 librados: calling monclient init 2024-03-29T11:04:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.896+0000 7f5e4a3c3640 10 monclient: init 2024-03-29T11:04:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.896+0000 7f5e4a3c3640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.896+0000 7f5e4a3c3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:28.877 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 10 monclient(hunting): picked mon.noname-b con 0x7f5e3c006800 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 10 monclient(hunting): picked mon.noname-a con 0x7f5e3c031b60 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 10 monclient(hunting): picked mon.noname-c con 0x7f5e3c0bd540 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c0c6180 con 0x7f5e3c031b60 2024-03-29T11:04:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c0c3e20 con 0x7f5e3c006800 2024-03-29T11:04:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c0c5d80 con 0x7f5e3c0bd540 2024-03-29T11:04:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:28.878 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.897+0000 7f5e4a3c3640 10 monclient(hunting): authenticate will time out at 1370.201554s 2024-03-29T11:04:28.886 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.899+0000 7f5e48bc0640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f5e3c031b60 0x7f5e3c0c6610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:59035/0 (socket says 172.21.15.22:59035) 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.899+0000 7f5e48bc0640 1 -- 172.21.15.22:0/3430214109 learned_addr learned my addr 172.21.15.22:0/3430214109 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 client.?.objecter ms_handle_connect 0x7f5e3c0bd540 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 client.?.objecter ms_handle_connect 0x7f5e3c031b60 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 client.?.objecter ms_handle_connect 0x7f5e3c006800 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4247270243 0 0) 0x7f5e3c0c5d80 con 0x7f5e3c0bd540 2024-03-29T11:04:28.887 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): my global_id is 4715 2024-03-29T11:04:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e28004eb0 con 0x7f5e3c0bd540 2024-03-29T11:04:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025947858 0 0) 0x7f5e3c0c3e20 con 0x7f5e3c006800 2024-03-29T11:04:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:28.888 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3c0c5d80 con 0x7f5e3c006800 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837842885 0 0) 0x7f5e3c0c6180 con 0x7f5e3c031b60 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e3c0c3e20 con 0x7f5e3c031b60 2024-03-29T11:04:28.889 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2237480076 0 0) 0x7f5e28004eb0 con 0x7f5e3c0bd540 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e28005620 con 0x7f5e3c0bd540 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5e40001750 con 0x7f5e3c0bd540 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): dump: 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.890 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:28.891 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.891 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:28.891 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:28.891 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 274646353 0 0) 0x7f5e3c0c5d80 con 0x7f5e3c006800 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e28004eb0 con 0x7f5e3c006800 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5e4404fdd0 con 0x7f5e3c006800 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): dump: 2024-03-29T11:04:28.892 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:28.893 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2741892690 0 0) 0x7f5e3c0c3e20 con 0x7f5e3c031b60 2024-03-29T11:04:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e3c0c5d80 con 0x7f5e3c031b60 2024-03-29T11:04:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5e34001630 con 0x7f5e3c031b60 2024-03-29T11:04:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:28.894 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient(hunting): dump: 2024-03-29T11:04:28.894 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.895 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3935557831 0 0) 0x7f5e28005620 con 0x7f5e3c0bd540 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 >> v1:172.21.15.26:6789/0 conn(0x7f5e3c006800 legacy=0x7f5e3c0bce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 >> v1:172.21.15.22:6789/0 conn(0x7f5e3c031b60 legacy=0x7f5e3c0c6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 monclient: found mon.c 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.900+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c0bd980 con 0x7f5e3c0bd540 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e39ffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e39ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:28.896 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 5 monclient: authenticate success, global_id 4715 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 10 monclient: _renew_subs 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c15d340 con 0x7f5e3c0bd540 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 10 monclient: _renew_subs 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c15c900 con 0x7f5e3c0bd540 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.901+0000 7f5e4a3c3640 1 librados: init done 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5e40001c60 con 0x7f5e3c0bd540 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e39ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f5e40003790 con 0x7f5e3c0bd540 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e39ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:28.897 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e39ffb640 10 monclient: dump: 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:28.898 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e4a3c3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e4a3c3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e4a3c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.902+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c004cd0 con 0x7f5e3c0bd540 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.903+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f5e40019760 con 0x7f5e3c0bd540 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.903+0000 7f5e3b7fe640 4 set_mon_vals no callback set 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.905+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7f5e40083650 con 0x7f5e3c0bd540 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.905+0000 7f5e39ffb640 10 client.4715.objecter ms_dispatch 0x7f5e44000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.905+0000 7f5e39ffb640 3 client.4715.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:04:28.899 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.905+0000 7f5e39ffb640 3 client.4715.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:04:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.905+0000 7f5e39ffb640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-29T11:04:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.906+0000 7f5e493c1640 10 monclient: get_auth_request con 0x7f5e2806e310 auth_method 0 2024-03-29T11:04:28.900 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.908+0000 7f5e39ffb640 10 client.4715.objecter ms_handle_connect 0x7f5e2806e310 2024-03-29T11:04:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.913+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5e4004ef90 con 0x7f5e3c0bd540 2024-03-29T11:04:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.913+0000 7f5e39ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:28.901 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:28.913+0000 7f5e39ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:29.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.245+0000 7f5e4a3c3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:29.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.245+0000 7f5e4a3c3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:29.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.245+0000 7f5e4a3c3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:29.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.245+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e3c0c5f10 con 0x7f5e3c0bd540 2024-03-29T11:04:29.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.246+0000 7f5e39ffb640 1 -- 172.21.15.22:0/3430214109 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7f5e40050ec0 con 0x7f5e3c0bd540 2024-03-29T11:04:29.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.246+0000 7f5e39ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:29.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.246+0000 7f5e39ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:29.227 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:29.227 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:29.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.250+0000 7f5e4a3c3640 10 librados: watch_flush enter 2024-03-29T11:04:29.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.250+0000 7f5e4a3c3640 10 librados: watch_flush exit 2024-03-29T11:04:29.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.250+0000 7f5e4a3c3640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-29T11:04:29.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.250+0000 7f5e4a3c3640 10 client.4715.objecter successfully canceled tick 2024-03-29T11:04:29.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.250+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 >> v1:172.21.15.22:6816/380221644 conn(0x7f5e2806e310 legacy=0x7f5e280707d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:29.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 10 monclient: shutdown 2024-03-29T11:04:29.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:29.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 >> v1:172.21.15.22:6790/0 conn(0x7f5e3c0bd540 legacy=0x7f5e3c0c6d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:29.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 shutdown_connections 2024-03-29T11:04:29.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 >> 172.21.15.22:0/3430214109 conn(0x7f5e3c0058f0 msgr2=0x7f5e3c038280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:29.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 shutdown_connections 2024-03-29T11:04:29.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 1 -- 172.21.15.22:0/3430214109 wait complete. 2024-03-29T11:04:29.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:29.251+0000 7f5e4a3c3640 1 librados: shutdown 2024-03-29T11:04:29.247 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:36.054 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:36.212 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.230+0000 7f6e75515640 10 monclient: get_monmap_and_config 2024-03-29T11:04:36.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.230+0000 7f6e75515640 10 monclient: build_initial_monmap 2024-03-29T11:04:36.213 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.230+0000 7f6e75515640 10 monclient: monmap: 2024-03-29T11:04:36.213 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:36.213 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:36.214 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:36.231735+0000 2024-03-29T11:04:36.214 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:36.231735+0000 2024-03-29T11:04:36.214 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:36.214 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:36.214 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:36.215 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:36.215 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:36.215 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:36.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.232+0000 7f6e75515640 1 Processor -- start 2024-03-29T11:04:36.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.232+0000 7f6e75515640 1 -- start start 2024-03-29T11:04:36.215 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.232+0000 7f6e75515640 10 monclient: init 2024-03-29T11:04:36.224 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:36.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:04:36.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 10 monclient(hunting): picked mon.noname-b con 0x7f6e700a9930 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:36.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 10 monclient(hunting): picked mon.noname-a con 0x7f6e70149dc0 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:36.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 10 monclient(hunting): picked mon.noname-c con 0x7f6e7014ff40 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:36.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e700aae60 con 0x7f6e70149dc0 2024-03-29T11:04:36.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e700b4310 con 0x7f6e7014ff40 2024-03-29T11:04:36.225 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e700afdd0 con 0x7f6e700a9930 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.233+0000 7f6e75515640 10 monclient(hunting): authenticate will time out at 1377.537509s 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.235+0000 7f6e6effd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f6e700a9930 0x7f6e70143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59030/0 (socket says 172.21.15.22:59030) 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.235+0000 7f6e6effd640 1 -- 172.21.15.22:0/1121495571 learned_addr learned my addr 172.21.15.22:0/1121495571 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.236+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602043437 0 0) 0x7f6e700afdd0 con 0x7f6e700a9930 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.236+0000 7f6e6dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.236+0000 7f6e6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.236+0000 7f6e6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:36.226 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.236+0000 7f6e6dffb640 10 monclient(hunting): my global_id is 4718 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.236+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e58003520 con 0x7f6e700a9930 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.237+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3563723564 0 0) 0x7f6e58003520 con 0x7f6e700a9930 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.237+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 >> v1:172.21.15.26:6789/0 conn(0x7f6e7014ff40 legacy=0x7f6e70152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.237+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 >> v1:172.21.15.22:6789/0 conn(0x7f6e70149dc0 legacy=0x7f6e7014a240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 1 monclient: found mon.noname-b 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7014a950 con 0x7f6e700a9930 2024-03-29T11:04:36.227 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6e60001670 con 0x7f6e700a9930 2024-03-29T11:04:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:36.228 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e6dffb640 10 monclient: dump: 2024-03-29T11:04:36.228 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:36.229 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e75515640 5 monclient: authenticate success, global_id 4718 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.238+0000 7f6e75515640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e600018d0 con 0x7f6e700a9930 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e75515640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e75515640 4 set_mon_vals no callback set 2024-03-29T11:04:36.230 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e6dffb640 1 -- 172.21.15.22:0/1121495571 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6e60001d20 con 0x7f6e700a9930 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e6dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.240+0000 7f6e6dffb640 10 monclient: dump: 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.231 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.241+0000 7f6e75515640 10 monclient: shutdown 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.241+0000 7f6e75515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:36.232 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.241+0000 7f6e75515640 1 -- 172.21.15.22:0/1121495571 >> v1:172.21.15.22:6790/0 conn(0x7f6e700a9930 legacy=0x7f6e70143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.241+0000 7f6e75515640 1 -- 172.21.15.22:0/1121495571 shutdown_connections 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.241+0000 7f6e75515640 1 -- 172.21.15.22:0/1121495571 >> 172.21.15.22:0/1121495571 conn(0x7f6e700b35e0 msgr2=0x7f6e700b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.241+0000 7f6e75515640 1 -- 172.21.15.22:0/1121495571 shutdown_connections 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.241+0000 7f6e75515640 1 -- 172.21.15.22:0/1121495571 wait complete. 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.242+0000 7f6e75515640 10 monclient: build_initial_monmap 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.242+0000 7f6e75515640 10 monclient: monmap: 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:36.233 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:36.234 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:36.234 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:36.234 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:36.234 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:36.234 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:36.234 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:36.234 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.243+0000 7f6e75515640 1 librados: starting msgr at 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.243+0000 7f6e75515640 1 librados: starting objecter 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.243+0000 7f6e75515640 1 Processor -- start 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.243+0000 7f6e75515640 1 -- start start 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.243+0000 7f6e75515640 1 librados: setting wanted keys 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.243+0000 7f6e75515640 1 librados: calling monclient init 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.243+0000 7f6e75515640 10 monclient: init 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.244+0000 7f6e75515640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:36.235 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.244+0000 7f6e75515640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:36.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 10 monclient(hunting): picked mon.noname-a con 0x7f6e700a9930 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:36.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 10 monclient(hunting): picked mon.noname-b con 0x7f6e70149dc0 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:36.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 10 monclient(hunting): picked mon.noname-c con 0x7f6e7014ff40 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:36.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7013d2b0 con 0x7f6e700a9930 2024-03-29T11:04:36.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e70144510 con 0x7f6e70149dc0 2024-03-29T11:04:36.239 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e7013d7f0 con 0x7f6e7014ff40 2024-03-29T11:04:36.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:36.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.246+0000 7f6e75515640 10 monclient(hunting): authenticate will time out at 1377.550459s 2024-03-29T11:04:36.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.251+0000 7f6e6e7fc640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f6e70149dc0 0x7f6e701deba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:47515/0 (socket says 172.21.15.22:47515) 2024-03-29T11:04:36.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.252+0000 7f6e6e7fc640 1 -- 172.21.15.22:0/999245334 learned_addr learned my addr 172.21.15.22:0/999245334 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:36.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 client.?.objecter ms_handle_connect 0x7f6e70149dc0 2024-03-29T11:04:36.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:36.240 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 client.?.objecter ms_handle_connect 0x7f6e700a9930 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 client.?.objecter ms_handle_connect 0x7f6e7014ff40 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274740290 0 0) 0x7f6e7013d2b0 con 0x7f6e700a9930 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:04:36.241 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e5c004eb0 con 0x7f6e700a9930 2024-03-29T11:04:36.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018412279 0 0) 0x7f6e70144510 con 0x7f6e70149dc0 2024-03-29T11:04:36.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:36.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:36.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:36.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:04:36.242 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e7013d2b0 con 0x7f6e70149dc0 2024-03-29T11:04:36.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034427105 0 0) 0x7f6e7013d7f0 con 0x7f6e7014ff40 2024-03-29T11:04:36.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _negotiate 2024-03-29T11:04:36.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:36.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:36.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 10 monclient(hunting): my global_id is 4721 2024-03-29T11:04:36.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.253+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e70144510 con 0x7f6e7014ff40 2024-03-29T11:04:36.243 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2891822314 0 0) 0x7f6e7013d2b0 con 0x7f6e70149dc0 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e7013d7f0 con 0x7f6e70149dc0 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 223018140 0 0) 0x7f6e70144510 con 0x7f6e7014ff40 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e7013d2b0 con 0x7f6e7014ff40 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6e68001be0 con 0x7f6e70149dc0 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 10 monclient(hunting): dump: 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:36.244 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:36.245 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.245 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.245 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:36.245 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:36.245 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:36.245 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:36.245 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6e64001610 con 0x7f6e7014ff40 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.256+0000 7f6e577fe640 10 monclient(hunting): dump: 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.246 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.247 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:36.247 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:36.247 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:36.247 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:36.247 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:36.247 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:36.247 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2558085915 0 0) 0x7f6e7013d7f0 con 0x7f6e70149dc0 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 >> v1:172.21.15.22:6790/0 conn(0x7f6e7014ff40 legacy=0x7f6e701df2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 >> v1:172.21.15.22:6789/0 conn(0x7f6e700a9930 legacy=0x7f6e7013a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: _finish_hunting 0 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 1 monclient: found mon.b 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7013d980 con 0x7f6e70149dc0 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: _finish_auth 0 2024-03-29T11:04:36.248 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: _check_auth_tickets 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 10 monclient: _renew_subs 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e70142530 con 0x7f6e70149dc0 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 10 monclient: _renew_subs 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:36.249 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e7013cde0 con 0x7f6e70149dc0 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e75515640 1 librados: init done 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e68001d40 con 0x7f6e70149dc0 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f6e68003b50 con 0x7f6e70149dc0 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e6d7fa640 4 set_mon_vals no callback set 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.257+0000 7f6e577fe640 10 monclient: dump: 2024-03-29T11:04:36.250 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:36.251 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.260+0000 7f6e75515640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.260+0000 7f6e75515640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.260+0000 7f6e75515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.260+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e50005080 con 0x7f6e70149dc0 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.262+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f6e68003db0 con 0x7f6e70149dc0 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.263+0000 7f6e6effd640 10 monclient: get_auth_request con 0x7f6e5c06dd00 auth_method 0 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.263+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7f6e68084c40 con 0x7f6e70149dc0 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.263+0000 7f6e577fe640 10 client.4510.objecter ms_dispatch 0x7f6e70000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.263+0000 7f6e577fe640 3 client.4510.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:04:36.253 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.263+0000 7f6e577fe640 3 client.4510.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:04:36.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.263+0000 7f6e577fe640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-29T11:04:36.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.263+0000 7f6e577fe640 10 client.4510.objecter ms_handle_connect 0x7f6e5c06dd00 2024-03-29T11:04:36.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.270+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e68050520 con 0x7f6e70149dc0 2024-03-29T11:04:36.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.270+0000 7f6e577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:36.254 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.270+0000 7f6e577fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:36.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.587+0000 7f6e75515640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:36.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.587+0000 7f6e75515640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:36.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.588+0000 7f6e75515640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:04:36.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.588+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e50005330 con 0x7f6e70149dc0 2024-03-29T11:04:36.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.589+0000 7f6e577fe640 1 -- 172.21.15.22:0/999245334 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7f6e68052450 con 0x7f6e70149dc0 2024-03-29T11:04:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.589+0000 7f6e577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:36.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.589+0000 7f6e577fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:36.570 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:36.571 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:36.572 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.593+0000 7f6e75515640 10 librados: watch_flush enter 2024-03-29T11:04:36.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 10 librados: watch_flush exit 2024-03-29T11:04:36.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-29T11:04:36.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 10 client.4510.objecter successfully canceled tick 2024-03-29T11:04:36.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 >> v1:172.21.15.22:6816/380221644 conn(0x7f6e5c06dd00 legacy=0x7f6e5c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:36.578 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 10 monclient: shutdown 2024-03-29T11:04:36.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:36.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 >> v1:172.21.15.26:6789/0 conn(0x7f6e70149dc0 legacy=0x7f6e701deba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:36.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 shutdown_connections 2024-03-29T11:04:36.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.596+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 >> 172.21.15.22:0/999245334 conn(0x7f6e700b35e0 msgr2=0x7f6e700b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:36.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.597+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 shutdown_connections 2024-03-29T11:04:36.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.597+0000 7f6e75515640 1 -- 172.21.15.22:0/999245334 wait complete. 2024-03-29T11:04:36.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:36.597+0000 7f6e75515640 1 librados: shutdown 2024-03-29T11:04:36.593 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:43.400 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.579+0000 7f15db59e640 10 monclient: get_monmap_and_config 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.579+0000 7f15db59e640 10 monclient: build_initial_monmap 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.579+0000 7f15db59e640 10 monclient: monmap: 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:43.580684+0000 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:43.580684+0000 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:43.559 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:43.560 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.581+0000 7f15db59e640 1 Processor -- start 2024-03-29T11:04:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.581+0000 7f15db59e640 1 -- start start 2024-03-29T11:04:43.560 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.581+0000 7f15db59e640 10 monclient: init 2024-03-29T11:04:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f15dc0b4820 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 10 monclient(hunting): picked mon.noname-a con 0x7f15dc0a9ce0 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 10 monclient(hunting): picked mon.noname-b con 0x7f15dc0b3cd0 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc0b4180 con 0x7f15dc0a9ce0 2024-03-29T11:04:43.561 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc0aaf60 con 0x7f15dc0b4820 2024-03-29T11:04:43.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc13c200 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:43.562 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.582+0000 7f15db59e640 10 monclient(hunting): authenticate will time out at 1384.886483s 2024-03-29T11:04:43.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.586+0000 7f15dad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f15dc0b3cd0 0x7f15dc14d7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:59402/0 (socket says 172.21.15.22:59402) 2024-03-29T11:04:43.565 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.586+0000 7f15dad9d640 1 -- 172.21.15.22:0/650761148 learned_addr learned my addr 172.21.15.22:0/650761148 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3647396572 0 0) 0x7f15dc0aaf60 con 0x7f15dc0b4820 2024-03-29T11:04:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:04:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:43.567 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15c4003520 con 0x7f15dc0b4820 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1940763886 0 0) 0x7f15dc13c200 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): _negotiate 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 10 monclient(hunting): my global_id is 4724 2024-03-29T11:04:43.568 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.587+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15dc0aaf60 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 428622430 0 0) 0x7f15dc0aaf60 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 >> v1:172.21.15.26:6789/0 conn(0x7f15dc0b4820 legacy=0x7f15dc13baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:43.569 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 >> v1:172.21.15.22:6789/0 conn(0x7f15dc0a9ce0 legacy=0x7f15dc145770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:43.570 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 10 monclient: _finish_hunting 0 2024-03-29T11:04:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 1 monclient: found mon.noname-b 2024-03-29T11:04:43.573 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc145e80 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 10 monclient: _finish_auth 0 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 10 monclient: _check_auth_tickets 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f15d40019f0 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 10 monclient: dump: 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.574 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15db59e640 5 monclient: authenticate success, global_id 4724 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15db59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.590+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15d4001b50 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.591+0000 7f15d959a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.591+0000 7f15d959a640 1 -- 172.21.15.22:0/650761148 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f15d4001f60 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.591+0000 7f15d959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.591+0000 7f15d959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.591+0000 7f15d959a640 10 monclient: dump: 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:43.575 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.591+0000 7f15db59e640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.591+0000 7f15db59e640 4 set_mon_vals no callback set 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.592+0000 7f15db59e640 10 monclient: shutdown 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.592+0000 7f15db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.592+0000 7f15db59e640 1 -- 172.21.15.22:0/650761148 >> v1:172.21.15.22:6790/0 conn(0x7f15dc0b3cd0 legacy=0x7f15dc14d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.592+0000 7f15db59e640 1 -- 172.21.15.22:0/650761148 shutdown_connections 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.592+0000 7f15db59e640 1 -- 172.21.15.22:0/650761148 >> 172.21.15.22:0/650761148 conn(0x7f15dc06d070 msgr2=0x7f15dc066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:43.576 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.594+0000 7f15db59e640 1 -- 172.21.15.22:0/650761148 shutdown_connections 2024-03-29T11:04:43.577 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.594+0000 7f15db59e640 1 -- 172.21.15.22:0/650761148 wait complete. 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.595+0000 7f15db59e640 10 monclient: build_initial_monmap 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.595+0000 7f15db59e640 10 monclient: monmap: 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:43.579 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.595+0000 7f15db59e640 1 librados: starting msgr at 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.595+0000 7f15db59e640 1 librados: starting objecter 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.596+0000 7f15db59e640 1 Processor -- start 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.596+0000 7f15db59e640 1 -- start start 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.596+0000 7f15db59e640 1 librados: setting wanted keys 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.596+0000 7f15db59e640 1 librados: calling monclient init 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.596+0000 7f15db59e640 10 monclient: init 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.597+0000 7f15db59e640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.597+0000 7f15db59e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.597+0000 7f15db59e640 10 monclient(hunting): picked mon.noname-a con 0x7f15dc0a9ce0 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.597+0000 7f15db59e640 10 monclient(hunting): picked mon.noname-b con 0x7f15dc0b3cd0 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15db59e640 10 monclient(hunting): picked mon.noname-c con 0x7f15dc0b4820 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:43.580 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15db59e640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc06fa40 con 0x7f15dc0a9ce0 2024-03-29T11:04:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15db59e640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc070b10 con 0x7f15dc0b3cd0 2024-03-29T11:04:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15db59e640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15dc0b15b0 con 0x7f15dc0b4820 2024-03-29T11:04:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15db59e640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15db59e640 10 monclient(hunting): authenticate will time out at 1384.902212s 2024-03-29T11:04:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15dad9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f15dc0b4820 0x7f15dc1d32a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35536/0 (socket says 172.21.15.22:35536) 2024-03-29T11:04:43.581 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.598+0000 7f15dad9d640 1 -- 172.21.15.22:0/848984176 learned_addr learned my addr 172.21.15.22:0/848984176 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:43.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 10 client.?.objecter ms_handle_connect 0x7f15dc0b4820 2024-03-29T11:04:43.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:43.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1047842158 0 0) 0x7f15dc0b15b0 con 0x7f15dc0b4820 2024-03-29T11:04:43.582 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 10 monclient(hunting): _negotiate 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 10 monclient(hunting): my global_id is 4727 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.600+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15bc004eb0 con 0x7f15dc0b4820 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.601+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925011015 0 0) 0x7f15bc004eb0 con 0x7f15dc0b4820 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.601+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15dc0b15b0 con 0x7f15dc0b4820 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.601+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f15d4001ee0 con 0x7f15dc0b4820 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.601+0000 7f15caffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.601+0000 7f15caffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.601+0000 7f15caffd640 10 monclient(hunting): dump: 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.583 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.601+0000 7f15caffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.602+0000 7f15caffd640 10 client.?.objecter ms_handle_connect 0x7f15dc0b3cd0 2024-03-29T11:04:43.584 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.602+0000 7f15caffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1878241740 0 0) 0x7f15dc0b15b0 con 0x7f15dc0b4820 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 >> v1:172.21.15.26:6789/0 conn(0x7f15dc0b3cd0 legacy=0x7f15dc1d2b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 >> v1:172.21.15.22:6789/0 conn(0x7f15dc0a9ce0 legacy=0x7f15dc06f200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 10 monclient: _finish_hunting 0 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 1 monclient: found mon.c 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.605+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc06e720 con 0x7f15dc0b4820 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.606+0000 7f15caffd640 10 monclient: _finish_auth 0 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.606+0000 7f15caffd640 10 monclient: _check_auth_tickets 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.606+0000 7f15caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:43.587 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.606+0000 7f15db59e640 5 monclient: authenticate success, global_id 4727 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.606+0000 7f15db59e640 10 monclient: _renew_subs 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.606+0000 7f15db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.606+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15dc1de3d0 con 0x7f15dc0b4820 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15db59e640 10 client.4727.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15db59e640 10 monclient: _renew_subs 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15dc1de740 con 0x7f15dc0b4820 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15db59e640 1 librados: init done 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f15d4004e10 con 0x7f15dc0b4820 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f15d4005260 con 0x7f15dc0b4820 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:43.588 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.607+0000 7f15d8d99640 4 set_mon_vals no callback set 2024-03-29T11:04:43.589 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.608+0000 7f15db59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.609+0000 7f15caffd640 10 monclient: dump: 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.609+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f15d40053c0 con 0x7f15dc0b4820 2024-03-29T11:04:43.590 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15db59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:43.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:43.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c0005080 con 0x7f15dc0b4820 2024-03-29T11:04:43.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7f15d4084fa0 con 0x7f15dc0b4820 2024-03-29T11:04:43.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15caffd640 10 client.4727.objecter ms_dispatch 0x7f15dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:04:43.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15caffd640 3 client.4727.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:04:43.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15caffd640 3 client.4727.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:04:43.591 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.610+0000 7f15caffd640 20 client.4727.objecter dump_active .. 0 homeless 2024-03-29T11:04:43.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.611+0000 7f15da59c640 10 monclient: get_auth_request con 0x7f15bc06dbe0 auth_method 0 2024-03-29T11:04:43.592 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.612+0000 7f15caffd640 10 client.4727.objecter ms_handle_connect 0x7f15bc06dbe0 2024-03-29T11:04:43.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.625+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15d4050880 con 0x7f15dc0b4820 2024-03-29T11:04:43.604 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.625+0000 7f15caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:43.605 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.625+0000 7f15caffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:43.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.997+0000 7f15db59e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:43.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.997+0000 7f15db59e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:43.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.997+0000 7f15db59e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:04:43.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:43.997+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15c0005330 con 0x7f15dc0b4820 2024-03-29T11:04:43.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.000+0000 7f15caffd640 1 -- 172.21.15.22:0/848984176 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7f15d40527b0 con 0x7f15dc0b4820 2024-03-29T11:04:43.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.000+0000 7f15caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:43.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.000+0000 7f15caffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:43.981 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:43.981 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:43.987 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.008+0000 7f15db59e640 10 librados: watch_flush enter 2024-03-29T11:04:43.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.008+0000 7f15db59e640 10 librados: watch_flush exit 2024-03-29T11:04:43.988 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.008+0000 7f15db59e640 20 client.4727.objecter shutdown clearing up homeless session... 2024-03-29T11:04:43.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.008+0000 7f15db59e640 10 client.4727.objecter successfully canceled tick 2024-03-29T11:04:43.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.009+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 >> v1:172.21.15.22:6816/380221644 conn(0x7f15bc06dbe0 legacy=0x7f15bc0700a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:43.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.009+0000 7f15db59e640 10 monclient: shutdown 2024-03-29T11:04:43.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.009+0000 7f15db59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:43.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.009+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 >> v1:172.21.15.22:6790/0 conn(0x7f15dc0b4820 legacy=0x7f15dc1d32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:43.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.009+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 shutdown_connections 2024-03-29T11:04:43.989 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.009+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 >> 172.21.15.22:0/848984176 conn(0x7f15dc06d070 msgr2=0x7f15dc0b71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:43.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.010+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 shutdown_connections 2024-03-29T11:04:43.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.010+0000 7f15db59e640 1 -- 172.21.15.22:0/848984176 wait complete. 2024-03-29T11:04:43.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:44.010+0000 7f15db59e640 1 librados: shutdown 2024-03-29T11:04:44.009 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:50.816 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:50.971 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.991+0000 7fbe2004a640 10 monclient: get_monmap_and_config 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.991+0000 7fbe2004a640 10 monclient: build_initial_monmap 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.991+0000 7fbe2004a640 10 monclient: monmap: 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:50.992651+0000 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:50.992651+0000 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:50.972 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:50.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.992+0000 7fbe2004a640 1 Processor -- start 2024-03-29T11:04:50.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.993+0000 7fbe2004a640 1 -- start start 2024-03-29T11:04:50.973 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.993+0000 7fbe2004a640 10 monclient: init 2024-03-29T11:04:50.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.993+0000 7fbe2004a640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:50.974 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.993+0000 7fbe2004a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.993+0000 7fbe2004a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe180a9ac0 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe2004a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe1813ee90 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe2004a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe180af900 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe2004a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe180afd10 con 0x7fbe180a9ac0 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe2004a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe180af6f0 con 0x7fbe1813ee90 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1e5e7640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fbe180af900 0x7fbe18149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:57742/0 (socket says 172.21.15.22:57742) 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1e5e7640 1 -- 172.21.15.22:0/2717772882 learned_addr learned my addr 172.21.15.22:0/2717772882 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe2004a640 1 -- 172.21.15.22:0/2717772882 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe180aac80 con 0x7fbe180af900 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe2004a640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe2004a640 10 monclient(hunting): authenticate will time out at 1392.298460s 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332466043 0 0) 0x7fbe180aac80 con 0x7fbe180af900 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 10 monclient(hunting): _negotiate 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:50.975 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 10 monclient(hunting): my global_id is 4730 2024-03-29T11:04:50.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe04003520 con 0x7fbe180af900 2024-03-29T11:04:50.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3558233463 0 0) 0x7fbe04003520 con 0x7fbe180af900 2024-03-29T11:04:50.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 >> v1:172.21.15.26:6789/0 conn(0x7fbe1813ee90 legacy=0x7fbe1813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:50.976 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.994+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 >> v1:172.21.15.22:6789/0 conn(0x7fbe180a9ac0 legacy=0x7fbe180adfe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: _finish_hunting 0 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 1 monclient: found mon.noname-b 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe180aaf60 con 0x7fbe180af900 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: _finish_auth 0 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: _check_auth_tickets 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbe10001a10 con 0x7fbe180af900 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:50.979 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: dump: 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe10001be0 con 0x7fbe180af900 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 1 -- 172.21.15.22:0/2717772882 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbe10002030 con 0x7fbe180af900 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:50.980 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe1cde4640 10 monclient: dump: 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe2004a640 5 monclient: authenticate success, global_id 4730 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe2004a640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.998+0000 7fbe2004a640 4 set_mon_vals no callback set 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.999+0000 7fbe2004a640 10 monclient: shutdown 2024-03-29T11:04:50.981 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.999+0000 7fbe2004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:50.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:50.999+0000 7fbe2004a640 1 -- 172.21.15.22:0/2717772882 >> v1:172.21.15.22:6790/0 conn(0x7fbe180af900 legacy=0x7fbe18149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:50.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.002+0000 7fbe2004a640 1 -- 172.21.15.22:0/2717772882 shutdown_connections 2024-03-29T11:04:50.982 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.002+0000 7fbe2004a640 1 -- 172.21.15.22:0/2717772882 >> 172.21.15.22:0/2717772882 conn(0x7fbe180b35e0 msgr2=0x7fbe180b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:50.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.006+0000 7fbe2004a640 1 -- 172.21.15.22:0/2717772882 shutdown_connections 2024-03-29T11:04:50.985 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.006+0000 7fbe2004a640 1 -- 172.21.15.22:0/2717772882 wait complete. 2024-03-29T11:04:50.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.007+0000 7fbe2004a640 10 monclient: build_initial_monmap 2024-03-29T11:04:50.986 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.007+0000 7fbe2004a640 10 monclient: monmap: 2024-03-29T11:04:50.986 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:50.987 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.008+0000 7fbe2004a640 1 librados: starting msgr at 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.008+0000 7fbe2004a640 1 librados: starting objecter 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.008+0000 7fbe2004a640 1 Processor -- start 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.008+0000 7fbe2004a640 1 -- start start 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.008+0000 7fbe2004a640 1 librados: setting wanted keys 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.008+0000 7fbe2004a640 1 librados: calling monclient init 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.008+0000 7fbe2004a640 10 monclient: init 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:04:50.990 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe180a9ac0 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:50.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe180af900 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:50.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe1813ee90 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:50.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe18071010 con 0x7fbe1813ee90 2024-03-29T11:04:50.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1813a590 con 0x7fbe180af900 2024-03-29T11:04:50.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe18072160 con 0x7fbe180a9ac0 2024-03-29T11:04:50.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:50.991 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.009+0000 7fbe2004a640 10 monclient(hunting): authenticate will time out at 1392.313324s 2024-03-29T11:04:50.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe1e5e7640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fbe1813ee90 0x7fbe18139e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:53751/0 (socket says 172.21.15.22:53751) 2024-03-29T11:04:50.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe1e5e7640 1 -- 172.21.15.22:0/3961839918 learned_addr learned my addr 172.21.15.22:0/3961839918 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:50.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 client.?.objecter ms_handle_connect 0x7fbe1813ee90 2024-03-29T11:04:50.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:50.993 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206367799 0 0) 0x7fbe18071010 con 0x7fbe1813ee90 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdfc004eb0 con 0x7fbe1813ee90 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 client.?.objecter ms_handle_connect 0x7fbe180a9ac0 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2908321315 0 0) 0x7fbdfc004eb0 con 0x7fbe1813ee90 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe18071010 con 0x7fbe1813ee90 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbe10001ef0 con 0x7fbe1813ee90 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.012+0000 7fbe0e7fc640 10 monclient(hunting): dump: 2024-03-29T11:04:50.994 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.013+0000 7fbe0e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.013+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3106740291 0 0) 0x7fbe18071010 con 0x7fbe1813ee90 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.013+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 >> v1:172.21.15.22:6790/0 conn(0x7fbe180a9ac0 legacy=0x7fbe1811b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:50.995 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.013+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 >> v1:172.21.15.26:6789/0 conn(0x7fbe180af900 legacy=0x7fbe18139770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:04:50.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:04:50.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 1 monclient: found mon.a 2024-03-29T11:04:50.997 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe18072440 con 0x7fbe1813ee90 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe2004a640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe2004a640 10 monclient: _renew_subs 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe2004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1811b360 con 0x7fbe1813ee90 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe10002050 con 0x7fbe1813ee90 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fbe10005150 con 0x7fbe1813ee90 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:50.998 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.015+0000 7fbe0e7fc640 10 monclient: dump: 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.016+0000 7fbe0ffff640 4 set_mon_vals no callback set 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.016+0000 7fbe2004a640 10 client.4545.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.016+0000 7fbe2004a640 10 monclient: _renew_subs 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.016+0000 7fbe2004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:50.999 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.016+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe180703a0 con 0x7fbe1813ee90 2024-03-29T11:04:51.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.016+0000 7fbe2004a640 1 librados: init done 2024-03-29T11:04:51.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.019+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fbe1001b120 con 0x7fbe1813ee90 2024-03-29T11:04:51.000 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.019+0000 7fbe2004a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.019+0000 7fbe2004a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.019+0000 7fbe2004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.019+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe18075f10 con 0x7fbe1813ee90 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.020+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7fbe10084f20 con 0x7fbe1813ee90 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.020+0000 7fbe0e7fc640 10 client.4545.objecter ms_dispatch 0x7fbe18000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.020+0000 7fbe0e7fc640 3 client.4545.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.020+0000 7fbe0e7fc640 3 client.4545.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.020+0000 7fbe0e7fc640 20 client.4545.objecter dump_active .. 0 homeless 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.020+0000 7fbe1dde6640 10 monclient: get_auth_request con 0x7fbdfc06dd00 auth_method 0 2024-03-29T11:04:51.001 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.020+0000 7fbe0e7fc640 10 client.4545.objecter ms_handle_connect 0x7fbdfc06dd00 2024-03-29T11:04:51.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.027+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe10050800 con 0x7fbe1813ee90 2024-03-29T11:04:51.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.027+0000 7fbe0e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:51.008 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.027+0000 7fbe0e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:51.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.355+0000 7fbe2004a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:51.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.355+0000 7fbe2004a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:51.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.355+0000 7fbe2004a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:51.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.355+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe1806ab70 con 0x7fbe1813ee90 2024-03-29T11:04:51.336 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:51.336 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:51.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.356+0000 7fbe0e7fc640 1 -- 172.21.15.22:0/3961839918 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7fbe10052730 con 0x7fbe1813ee90 2024-03-29T11:04:51.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.356+0000 7fbe0e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:51.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.356+0000 7fbe0e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:51.341 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.362+0000 7fbe2004a640 10 librados: watch_flush enter 2024-03-29T11:04:51.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.363+0000 7fbe2004a640 10 librados: watch_flush exit 2024-03-29T11:04:51.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.363+0000 7fbe2004a640 20 client.4545.objecter shutdown clearing up homeless session... 2024-03-29T11:04:51.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.363+0000 7fbe2004a640 10 client.4545.objecter successfully canceled tick 2024-03-29T11:04:51.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.363+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 >> v1:172.21.15.22:6816/380221644 conn(0x7fbdfc06dd00 legacy=0x7fbdfc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:51.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.363+0000 7fbe2004a640 10 monclient: shutdown 2024-03-29T11:04:51.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.363+0000 7fbe2004a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:51.342 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.363+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 >> v1:172.21.15.22:6789/0 conn(0x7fbe1813ee90 legacy=0x7fbe18139e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:51.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.364+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 shutdown_connections 2024-03-29T11:04:51.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.364+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 >> 172.21.15.22:0/3961839918 conn(0x7fbe180b35e0 msgr2=0x7fbe181ccf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:51.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.364+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 shutdown_connections 2024-03-29T11:04:51.343 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.364+0000 7fbe2004a640 1 -- 172.21.15.22:0/3961839918 wait complete. 2024-03-29T11:04:51.344 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:51.364+0000 7fbe2004a640 1 librados: shutdown 2024-03-29T11:04:51.362 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:04:58.169 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.333+0000 7fa53fa50640 10 monclient: get_monmap_and_config 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.333+0000 7fa53fa50640 10 monclient: build_initial_monmap 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.333+0000 7fa53fa50640 10 monclient: monmap: 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:04:58.335052+0000 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:04:58.335052+0000 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:58.313 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:58.314 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:58.314 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:04:58.314 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:04:58.314 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.334+0000 7fa53fa50640 1 Processor -- start 2024-03-29T11:04:58.314 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.334+0000 7fa53fa50640 1 -- start start 2024-03-29T11:04:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.334+0000 7fa53fa50640 10 monclient: init 2024-03-29T11:04:58.315 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 10 monclient(hunting): picked mon.noname-c con 0x7fa5381422b0 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 10 monclient(hunting): picked mon.noname-a con 0x7fa538142690 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 10 monclient(hunting): picked mon.noname-b con 0x7fa5380b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa538142c80 con 0x7fa538142690 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5380aae40 con 0x7fa5381422b0 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53813fbd0 con 0x7fa5380b17d0 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53fa50640 10 monclient(hunting): authenticate will time out at 1399.640732s 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53dfed640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7fa5380b17d0 0x7fa5380b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:35972/0 (socket says 172.21.15.22:35972) 2024-03-29T11:04:58.316 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.335+0000 7fa53dfed640 1 -- 172.21.15.22:0/1330020354 learned_addr learned my addr 172.21.15.22:0/1330020354 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.336+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162362853 0 0) 0x7fa53813fbd0 con 0x7fa5380b17d0 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.336+0000 7fa52ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.336+0000 7fa52ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.336+0000 7fa52ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.336+0000 7fa52ffff640 10 monclient(hunting): my global_id is 4736 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.336+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa524003520 con 0x7fa5380b17d0 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245620763 0 0) 0x7fa538142c80 con 0x7fa538142690 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:04:58.318 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53813fbd0 con 0x7fa538142690 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3464341451 0 0) 0x7fa524003520 con 0x7fa5380b17d0 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 >> v1:172.21.15.26:6789/0 conn(0x7fa5381422b0 legacy=0x7fa53813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 >> v1:172.21.15.22:6789/0 conn(0x7fa538142690 legacy=0x7fa5380b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 monclient: found mon.noname-b 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.22:6790/0 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa538151940 con 0x7fa5380b17d0 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: _finish_auth 0 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa530001a10 con 0x7fa5380b17d0 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:04:58.319 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: dump: 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa53fa50640 5 monclient: authenticate success, global_id 4736 2024-03-29T11:04:58.320 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa53fa50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:04:58.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 <== mon.2 v1:172.21.15.22:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa530001b70 con 0x7fa5380b17d0 2024-03-29T11:04:58.328 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa53fa50640 10 monclient: get_monmap_and_config success 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 1 -- 172.21.15.22:0/1330020354 <== mon.2 v1:172.21.15.22:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa530001f00 con 0x7fa5380b17d0 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa53fa50640 4 set_mon_vals no callback set 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.337+0000 7fa52ffff640 10 monclient: dump: 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:58.329 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.338+0000 7fa53fa50640 10 monclient: shutdown 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.338+0000 7fa53fa50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.338+0000 7fa53fa50640 1 -- 172.21.15.22:0/1330020354 >> v1:172.21.15.22:6790/0 conn(0x7fa5380b17d0 legacy=0x7fa5380b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.339+0000 7fa53fa50640 1 -- 172.21.15.22:0/1330020354 shutdown_connections 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.339+0000 7fa53fa50640 1 -- 172.21.15.22:0/1330020354 >> 172.21.15.22:0/1330020354 conn(0x7fa538066a00 msgr2=0x7fa538066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.339+0000 7fa53fa50640 1 -- 172.21.15.22:0/1330020354 shutdown_connections 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.339+0000 7fa53fa50640 1 -- 172.21.15.22:0/1330020354 wait complete. 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 10 monclient: build_initial_monmap 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 10 monclient: monmap: 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:04:58.330 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 1 librados: starting msgr at 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 1 librados: starting objecter 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 1 Processor -- start 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 1 -- start start 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 1 librados: setting wanted keys 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 1 librados: calling monclient init 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.340+0000 7fa53fa50640 10 monclient: init 2024-03-29T11:04:58.331 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 10 monclient: _reopen_session rank -1 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 10 monclient(hunting): picked mon.noname-c con 0x7fa5380b17d0 addr v1:172.21.15.22:6790/0 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 10 monclient(hunting): picked mon.noname-a con 0x7fa5381422b0 addr v1:172.21.15.22:6789/0 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 10 monclient(hunting): picked mon.noname-b con 0x7fa538142690 addr v1:172.21.15.26:6789/0 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53806f830 con 0x7fa5381422b0 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa538071620 con 0x7fa538142690 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa53806ef50 con 0x7fa5380b17d0 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 10 monclient(hunting): _renew_subs 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53fa50640 10 monclient(hunting): authenticate will time out at 1399.646834s 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53cfeb640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa5381422b0 0x7fa53813beb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:44086/0 (socket says 172.21.15.22:44086) 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa53cfeb640 1 -- 172.21.15.22:0/1189853964 learned_addr learned my addr 172.21.15.22:0/1189853964 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa52dffb640 10 client.?.objecter ms_handle_connect 0x7fa5381422b0 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa52dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:58.332 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa52dffb640 10 client.?.objecter ms_handle_connect 0x7fa5380b17d0 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.341+0000 7fa52dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282706407 0 0) 0x7fa53806f830 con 0x7fa5381422b0 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c004eb0 con 0x7fa5381422b0 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822478814 0 0) 0x7fa53806ef50 con 0x7fa5380b17d0 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 10 monclient(hunting): my global_id is 4739 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.342+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa53806f830 con 0x7fa5380b17d0 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 349444375 0 0) 0x7fa53806f830 con 0x7fa5380b17d0 2024-03-29T11:04:58.333 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa53806ef50 con 0x7fa5380b17d0 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2303186974 0 0) 0x7fa51c004eb0 con 0x7fa5381422b0 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa53806f830 con 0x7fa5381422b0 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa528001930 con 0x7fa5380b17d0 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient(hunting): dump: 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:58.334 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa534001910 con 0x7fa5381422b0 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient(hunting): dump: 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:58.335 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 client.?.objecter ms_handle_connect 0x7fa538142690 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 506826050 0 0) 0x7fa53806f830 con 0x7fa5381422b0 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 >> v1:172.21.15.22:6790/0 conn(0x7fa5380b17d0 legacy=0x7fa53806fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 >> v1:172.21.15.26:6789/0 conn(0x7fa538142690 legacy=0x7fa5381e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 monclient: found mon.a 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5380703d0 con 0x7fa5381422b0 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient: _finish_auth 0 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:04:58.336 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa52dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa53fa50640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa53fa50640 10 monclient: _renew_subs 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa53fa50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.343+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa538070620 con 0x7fa5381422b0 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa53fa50640 10 client.4551.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa53fa50640 10 monclient: _renew_subs 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa53fa50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5381deff0 con 0x7fa5381422b0 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa53fa50640 1 librados: init done 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5340035f0 con 0x7fa5381422b0 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa52dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa534001a70 con 0x7fa5381422b0 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa52dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa52dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa52effd640 4 set_mon_vals no callback set 2024-03-29T11:04:58.337 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.344+0000 7fa52dffb640 10 monclient: dump: 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.346+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fa5340037e0 con 0x7fa5381422b0 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.347+0000 7fa53fa50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.347+0000 7fa53fa50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:58.338 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.347+0000 7fa53fa50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.347+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa518005080 con 0x7fa5381422b0 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.347+0000 7fa53d7ec640 10 monclient: get_auth_request con 0x7fa51c06de10 auth_method 0 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.348+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7fa5340847b0 con 0x7fa5381422b0 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.348+0000 7fa52dffb640 10 client.4551.objecter ms_dispatch 0x7fa538000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.348+0000 7fa52dffb640 3 client.4551.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.348+0000 7fa52dffb640 3 client.4551.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.348+0000 7fa52dffb640 20 client.4551.objecter dump_active .. 0 homeless 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.348+0000 7fa52dffb640 10 client.4551.objecter ms_handle_connect 0x7fa51c06de10 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.359+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa534050090 con 0x7fa5381422b0 2024-03-29T11:04:58.339 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.359+0000 7fa52dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:04:58.340 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.359+0000 7fa52dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:04:58.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.729+0000 7fa53fa50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:58.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.729+0000 7fa53fa50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:58.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.729+0000 7fa53fa50640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:04:58.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.729+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa518005330 con 0x7fa5381422b0 2024-03-29T11:04:58.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.731+0000 7fa52dffb640 1 -- 172.21.15.22:0/1189853964 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7fa534051fc0 con 0x7fa5381422b0 2024-03-29T11:04:58.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.731+0000 7fa52dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:04:58.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.731+0000 7fa52dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:04:58.712 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:04:58.712 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:04:58.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 10 librados: watch_flush enter 2024-03-29T11:04:58.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 10 librados: watch_flush exit 2024-03-29T11:04:58.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 20 client.4551.objecter shutdown clearing up homeless session... 2024-03-29T11:04:58.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 10 client.4551.objecter successfully canceled tick 2024-03-29T11:04:58.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 >> v1:172.21.15.22:6816/380221644 conn(0x7fa51c06de10 legacy=0x7fa51c0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:58.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 10 monclient: shutdown 2024-03-29T11:04:58.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:04:58.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.734+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 >> v1:172.21.15.22:6789/0 conn(0x7fa5381422b0 legacy=0x7fa53813beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:04:58.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.735+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 shutdown_connections 2024-03-29T11:04:58.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.735+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 >> 172.21.15.22:0/1189853964 conn(0x7fa538066a00 msgr2=0x7fa53814dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:04:58.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.735+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 shutdown_connections 2024-03-29T11:04:58.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.735+0000 7fa53fa50640 1 -- 172.21.15.22:0/1189853964 wait complete. 2024-03-29T11:04:58.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:04:58.735+0000 7fa53fa50640 1 librados: shutdown 2024-03-29T11:04:58.729 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:05.537 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.711+0000 7f29aca9c640 10 monclient: get_monmap_and_config 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.711+0000 7f29aca9c640 10 monclient: build_initial_monmap 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.711+0000 7f29aca9c640 10 monclient: monmap: 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:05:05.713552+0000 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:05:05.713552+0000 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:05:05.693 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:05.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.712+0000 7f29aca9c640 1 Processor -- start 2024-03-29T11:05:05.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.712+0000 7f29aca9c640 1 -- start start 2024-03-29T11:05:05.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.712+0000 7f29aca9c640 10 monclient: init 2024-03-29T11:05:05.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:05.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:05:05.694 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 10 monclient(hunting): picked mon.noname-a con 0x7f29a80ae250 addr v1:172.21.15.22:6789/0 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 10 monclient(hunting): picked mon.noname-c con 0x7f29a813f970 addr v1:172.21.15.26:6789/0 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 10 monclient(hunting): picked mon.noname-b con 0x7f29a8142800 addr v1:172.21.15.22:6790/0 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a80afe00 con 0x7f29a80ae250 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a8142c10 con 0x7f29a813f970 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a80afb20 con 0x7f29a8142800 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29aca9c640 10 monclient(hunting): authenticate will time out at 1407.019528s 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29a6d9d640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f29a8142800 0x7f29a814da10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:52968/0 (socket says 172.21.15.22:52968) 2024-03-29T11:05:05.695 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.714+0000 7f29a6d9d640 1 -- 172.21.15.22:0/2073895726 learned_addr learned my addr 172.21.15.22:0/2073895726 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797976884 0 0) 0x7f29a8142c10 con 0x7f29a813f970 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): _negotiate 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2990003520 con 0x7f29a813f970 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 212101283 0 0) 0x7f29a80afb20 con 0x7f29a8142800 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): _negotiate 2024-03-29T11:05:05.698 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient(hunting): my global_id is 4742 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29a8142c10 con 0x7f29a8142800 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3710590226 0 0) 0x7f2990003520 con 0x7f29a813f970 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 >> v1:172.21.15.22:6790/0 conn(0x7f29a8142800 legacy=0x7f29a814da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 >> v1:172.21.15.22:6789/0 conn(0x7f29a80ae250 legacy=0x7f29a813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 10 monclient: _finish_hunting 0 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 1 monclient: found mon.noname-c 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.715+0000 7f29a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a80aab80 con 0x7f29a813f970 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: _finish_auth 0 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: _check_auth_tickets 2024-03-29T11:05:05.699 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29aca9c640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29aca9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f299c0015c0 con 0x7f29a813f970 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: dump: 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:05.700 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29aca9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f299c0017e0 con 0x7f29a813f970 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29aca9c640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29aca9c640 4 set_mon_vals no callback set 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 1 -- 172.21.15.22:0/2073895726 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f299c001b70 con 0x7f29a813f970 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.716+0000 7f29a559a640 10 monclient: dump: 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:05.701 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.717+0000 7f29aca9c640 10 monclient: shutdown 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.717+0000 7f29aca9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.717+0000 7f29aca9c640 1 -- 172.21.15.22:0/2073895726 >> v1:172.21.15.26:6789/0 conn(0x7f29a813f970 legacy=0x7f29a81420f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.717+0000 7f29aca9c640 1 -- 172.21.15.22:0/2073895726 shutdown_connections 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.717+0000 7f29aca9c640 1 -- 172.21.15.22:0/2073895726 >> 172.21.15.22:0/2073895726 conn(0x7f29a80b35e0 msgr2=0x7f29a80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.717+0000 7f29aca9c640 1 -- 172.21.15.22:0/2073895726 shutdown_connections 2024-03-29T11:05:05.702 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.717+0000 7f29aca9c640 1 -- 172.21.15.22:0/2073895726 wait complete. 2024-03-29T11:05:05.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.718+0000 7f29aca9c640 10 monclient: build_initial_monmap 2024-03-29T11:05:05.709 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.718+0000 7f29aca9c640 10 monclient: monmap: 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.719+0000 7f29aca9c640 1 librados: starting msgr at 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.719+0000 7f29aca9c640 1 librados: starting objecter 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.719+0000 7f29aca9c640 1 Processor -- start 2024-03-29T11:05:05.710 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.719+0000 7f29aca9c640 1 -- start start 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.719+0000 7f29aca9c640 1 librados: setting wanted keys 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.719+0000 7f29aca9c640 1 librados: calling monclient init 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.719+0000 7f29aca9c640 10 monclient: init 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 10 monclient(hunting): picked mon.noname-b con 0x7f29a80ae250 addr v1:172.21.15.26:6789/0 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 10 monclient(hunting): picked mon.noname-a con 0x7f29a813f970 addr v1:172.21.15.22:6789/0 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 10 monclient(hunting): picked mon.noname-c con 0x7f29a8142800 addr v1:172.21.15.22:6790/0 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a8146170 con 0x7f29a813f970 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a81db930 con 0x7f29a80ae250 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a81385f0 con 0x7f29a8142800 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.720+0000 7f29aca9c640 10 monclient(hunting): authenticate will time out at 1407.026076s 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f29a659c640 1 --1- >> v1:172.21.15.26:6789/0 conn(0x7f29a80ae250 0x7f29a8137ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.26:6789/0 says I am v1:172.21.15.22:50354/0 (socket says 172.21.15.22:50354) 2024-03-29T11:05:05.711 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f29a659c640 1 -- 172.21.15.22:0/3342323318 learned_addr learned my addr 172.21.15.22:0/3342323318 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 10 client.?.objecter ms_handle_connect 0x7f29a80ae250 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539980598 0 0) 0x7f29a81db930 con 0x7f29a80ae250 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.721+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2988004eb0 con 0x7f29a80ae250 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1951828546 0 0) 0x7f2988004eb0 con 0x7f29a80ae250 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29a81db930 con 0x7f29a80ae250 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2998001a30 con 0x7f29a80ae250 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:05:05.712 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient(hunting): dump: 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 client.?.objecter ms_handle_connect 0x7f29a813f970 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4118262814 0 0) 0x7f29a81db930 con 0x7f29a80ae250 2024-03-29T11:05:05.713 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 >> v1:172.21.15.22:6790/0 conn(0x7f29a8142800 legacy=0x7f29a8144440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 >> v1:172.21.15.22:6789/0 conn(0x7f29a813f970 legacy=0x7f29a81dbe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 monclient: found mon.b 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a81dc560 con 0x7f29a80ae250 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: _finish_auth 0 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 5 monclient: authenticate success, global_id 4528 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 10 monclient: _renew_subs 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:05:05.714 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 --> v1:172.21.15.26:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a8143b10 con 0x7f29a80ae250 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 10 client.4528.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 10 monclient: _renew_subs 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 --> v1:172.21.15.26:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a813a7c0 con 0x7f29a80ae250 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f29aca9c640 1 librados: init done 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2998001d30 con 0x7f29a80ae250 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f2998003a60 con 0x7f29a80ae250 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.722+0000 7f2996ffd640 10 monclient: dump: 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:05.715 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.724+0000 7f29aca9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.724+0000 7f29aca9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.724+0000 7f29aca9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.724+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a81dbac0 con 0x7f29a80ae250 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.724+0000 7f29a4d99640 4 set_mon_vals no callback set 2024-03-29T11:05:05.716 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.724+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f2998003bc0 con 0x7f29a80ae250 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.725+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7f2998084c10 con 0x7f29a80ae250 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.725+0000 7f2996ffd640 10 client.4528.objecter ms_dispatch 0x7f29a8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.725+0000 7f2996ffd640 3 client.4528.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.725+0000 7f2996ffd640 3 client.4528.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.725+0000 7f2996ffd640 20 client.4528.objecter dump_active .. 0 homeless 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.726+0000 7f29a5d9b640 10 monclient: get_auth_request con 0x7f298806dc20 auth_method 0 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.726+0000 7f2996ffd640 10 client.4528.objecter ms_handle_connect 0x7f298806dc20 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.732+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2998050320 con 0x7f29a80ae250 2024-03-29T11:05:05.717 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.732+0000 7f2996ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:05.718 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:05.732+0000 7f2996ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:06.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.146+0000 7f29aca9c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:06.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.146+0000 7f29aca9c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:06.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.146+0000 7f29aca9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.26:6789/0 2024-03-29T11:05:06.126 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.146+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 --> v1:172.21.15.26:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29a8075f10 con 0x7f29a80ae250 2024-03-29T11:05:06.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.148+0000 7f2996ffd640 1 -- 172.21.15.22:0/3342323318 <== mon.1 v1:172.21.15.26:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7f2998052420 con 0x7f29a80ae250 2024-03-29T11:05:06.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.148+0000 7f2996ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:06.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.148+0000 7f2996ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:06.128 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:05:06.128 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:06.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 10 librados: watch_flush enter 2024-03-29T11:05:06.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 10 librados: watch_flush exit 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 20 client.4528.objecter shutdown clearing up homeless session... 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 10 client.4528.objecter successfully canceled tick 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 >> v1:172.21.15.22:6816/380221644 conn(0x7f298806dc20 legacy=0x7f29880700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 10 monclient: shutdown 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 >> v1:172.21.15.26:6789/0 conn(0x7f29a80ae250 legacy=0x7f29a8137ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 shutdown_connections 2024-03-29T11:05:06.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.152+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 >> 172.21.15.22:0/3342323318 conn(0x7f29a80b35e0 msgr2=0x7f29a80aa1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:06.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.153+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 shutdown_connections 2024-03-29T11:05:06.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.153+0000 7f29aca9c640 1 -- 172.21.15.22:0/3342323318 wait complete. 2024-03-29T11:05:06.134 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:06.153+0000 7f29aca9c640 1 librados: shutdown 2024-03-29T11:05:06.148 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:12.957 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:13.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.134+0000 7f437c8ac640 10 monclient: get_monmap_and_config 2024-03-29T11:05:13.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.134+0000 7f437c8ac640 10 monclient: build_initial_monmap 2024-03-29T11:05:13.114 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.134+0000 7f437c8ac640 10 monclient: monmap: 2024-03-29T11:05:13.114 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:05:13.114 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:05:13.135587+0000 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:05:13.135587+0000 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:05:13.115 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:13.117 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.138+0000 7f437c8ac640 1 Processor -- start 2024-03-29T11:05:13.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.138+0000 7f437c8ac640 1 -- start start 2024-03-29T11:05:13.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.138+0000 7f437c8ac640 10 monclient: init 2024-03-29T11:05:13.121 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 10 monclient(hunting): picked mon.noname-a con 0x7f4370006a70 addr v1:172.21.15.22:6789/0 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 10 monclient(hunting): picked mon.noname-b con 0x7f437002b100 addr v1:172.21.15.22:6790/0 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 10 monclient(hunting): picked mon.noname-c con 0x7f4370029c30 addr v1:172.21.15.26:6789/0 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437002a270 con 0x7f4370006a70 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437002c440 con 0x7f4370029c30 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700c9440 con 0x7f437002b100 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:13.122 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.139+0000 7f437c8ac640 10 monclient(hunting): authenticate will time out at 1414.443704s 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.142+0000 7f4376ffd640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f437002b100 0x7f43700cf100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:36515/0 (socket says 172.21.15.22:36515) 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.142+0000 7f4376ffd640 1 -- 172.21.15.22:0/807116472 learned_addr learned my addr 172.21.15.22:0/807116472 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615559533 0 0) 0x7f437002c440 con 0x7f4370029c30 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient(hunting): _negotiate 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4368003520 con 0x7f4370029c30 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431694856 0 0) 0x7f4368003520 con 0x7f4370029c30 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 >> v1:172.21.15.22:6790/0 conn(0x7f437002b100 legacy=0x7f43700cf100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 >> v1:172.21.15.22:6789/0 conn(0x7f4370006a70 legacy=0x7f43700c8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient: _finish_hunting 0 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 1 monclient: found mon.noname-c 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:13.127 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.26:6789/0 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 --> v1:172.21.15.26:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43700c95d0 con 0x7f4370029c30 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient: _finish_auth 0 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 10 monclient: _check_auth_tickets 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.146+0000 7f43767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f437c8ac640 5 monclient: authenticate success, global_id 4531 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f437c8ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4364001690 con 0x7f4370029c30 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f43767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f43767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f43767fc640 10 monclient: dump: 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:13.128 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f437c8ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 <== mon.1 v1:172.21.15.26:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43640017f0 con 0x7f4370029c30 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f43767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f437c8ac640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.147+0000 7f437c8ac640 4 set_mon_vals no callback set 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f43767fc640 1 -- 172.21.15.22:0/807116472 <== mon.1 v1:172.21.15.26:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4364001c40 con 0x7f4370029c30 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f43767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f43767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f43767fc640 10 monclient: dump: 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.129 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f437c8ac640 10 monclient: shutdown 2024-03-29T11:05:13.130 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f437c8ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f437c8ac640 1 -- 172.21.15.22:0/807116472 >> v1:172.21.15.26:6789/0 conn(0x7f4370029c30 legacy=0x7f43700d74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f437c8ac640 1 -- 172.21.15.22:0/807116472 shutdown_connections 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f437c8ac640 1 -- 172.21.15.22:0/807116472 >> 172.21.15.22:0/807116472 conn(0x7f43700058f0 msgr2=0x7f4370005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f437c8ac640 1 -- 172.21.15.22:0/807116472 shutdown_connections 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.148+0000 7f437c8ac640 1 -- 172.21.15.22:0/807116472 wait complete. 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.149+0000 7f437c8ac640 10 monclient: build_initial_monmap 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.149+0000 7f437c8ac640 10 monclient: monmap: 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:05:13.131 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.150+0000 7f437c8ac640 1 librados: starting msgr at 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.150+0000 7f437c8ac640 1 librados: starting objecter 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.150+0000 7f437c8ac640 1 Processor -- start 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.150+0000 7f437c8ac640 1 -- start start 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.150+0000 7f437c8ac640 1 librados: setting wanted keys 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.150+0000 7f437c8ac640 1 librados: calling monclient init 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.150+0000 7f437c8ac640 10 monclient: init 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 10 monclient(hunting): picked mon.noname-c con 0x7f4370006a70 addr v1:172.21.15.22:6790/0 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 10 monclient(hunting): picked mon.noname-a con 0x7f4370029c30 addr v1:172.21.15.22:6789/0 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 10 monclient(hunting): picked mon.noname-b con 0x7f4370031c20 addr v1:172.21.15.26:6789/0 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437002d940 con 0x7f4370029c30 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 1 -- --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f437002daf0 con 0x7f4370031c20 2024-03-29T11:05:13.132 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 1 -- --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43700326c0 con 0x7f4370006a70 2024-03-29T11:05:13.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:13.133 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.151+0000 7f437c8ac640 10 monclient(hunting): authenticate will time out at 1414.455522s 2024-03-29T11:05:13.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.156+0000 7f43777fe640 1 --1- >> v1:172.21.15.22:6790/0 conn(0x7f4370006a70 0x7f4370160de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6790/0 says I am v1:172.21.15.22:36561/0 (socket says 172.21.15.22:36561) 2024-03-29T11:05:13.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.156+0000 7f43777fe640 1 -- 172.21.15.22:0/1126417872 learned_addr learned my addr 172.21.15.22:0/1126417872 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:05:13.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 10 client.?.objecter ms_handle_connect 0x7f4370006a70 2024-03-29T11:05:13.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:13.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 10 client.?.objecter ms_handle_connect 0x7f4370031c20 2024-03-29T11:05:13.139 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688598035 0 0) 0x7f43700326c0 con 0x7f4370006a70 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.158+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): my global_id is 4745 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f435c004eb0 con 0x7f4370006a70 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.1 v1:172.21.15.26:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2612170746 0 0) 0x7f437002daf0 con 0x7f4370031c20 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.26:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43700326c0 con 0x7f4370031c20 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480845279 0 0) 0x7f435c004eb0 con 0x7f4370006a70 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.22:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f437002daf0 con 0x7f4370006a70 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f436c001a30 con 0x7f4370006a70 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:13.140 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): dump: 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.159+0000 7f4363fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.1 v1:172.21.15.26:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729426804 0 0) 0x7f43700326c0 con 0x7f4370031c20 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.26:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f435c004eb0 con 0x7f4370031c20 2024-03-29T11:05:13.141 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.1 v1:172.21.15.26:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f4364001b00 con 0x7f4370031c20 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 10 monclient(hunting): dump: 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1955846828 0 0) 0x7f437002daf0 con 0x7f4370006a70 2024-03-29T11:05:13.142 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 >> v1:172.21.15.26:6789/0 conn(0x7f4370031c20 legacy=0x7f43700c2610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.160+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 >> v1:172.21.15.22:6789/0 conn(0x7f4370029c30 legacy=0x7f437002d230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 1 monclient: found mon.c 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.22:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4370032850 con 0x7f4370006a70 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 10 monclient: _finish_auth 0 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f4363fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 5 monclient: authenticate success, global_id 4745 2024-03-29T11:05:13.143 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 10 monclient: _renew_subs 2024-03-29T11:05:13.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:05:13.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.22:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43700c2d20 con 0x7f4370006a70 2024-03-29T11:05:13.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 10 client.4745.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:13.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 10 monclient: _renew_subs 2024-03-29T11:05:13.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:05:13.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.22:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f437002e010 con 0x7f4370006a70 2024-03-29T11:05:13.144 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.161+0000 7f437c8ac640 1 librados: init done 2024-03-29T11:05:13.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.165+0000 7f437c8ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:13.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.165+0000 7f437c8ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:13.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.165+0000 7f437c8ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:05:13.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.165+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f437002dd30 con 0x7f4370006a70 2024-03-29T11:05:13.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.165+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f436c001c10 con 0x7f4370006a70 2024-03-29T11:05:13.145 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.165+0000 7f4363fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.165+0000 7f4375ffb640 4 set_mon_vals no callback set 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.166+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7f436c003a60 con 0x7f4370006a70 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.166+0000 7f4363fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.166+0000 7f4363fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.166+0000 7f4363fff640 10 monclient: dump: 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:13.146 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:13.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.167+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7f436c003bc0 con 0x7f4370006a70 2024-03-29T11:05:13.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.168+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7f436c084a40 con 0x7f4370006a70 2024-03-29T11:05:13.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.168+0000 7f4363fff640 10 client.4745.objecter ms_dispatch 0x7f4378000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:05:13.147 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.168+0000 7f4363fff640 3 client.4745.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:05:13.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.168+0000 7f4363fff640 3 client.4745.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:05:13.148 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.168+0000 7f4363fff640 20 client.4745.objecter dump_active .. 0 homeless 2024-03-29T11:05:13.150 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.171+0000 7f4376ffd640 10 monclient: get_auth_request con 0x7f435c06de40 auth_method 0 2024-03-29T11:05:13.153 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.175+0000 7f4363fff640 10 client.4745.objecter ms_handle_connect 0x7f435c06de40 2024-03-29T11:05:13.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.176+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f436c050320 con 0x7f4370006a70 2024-03-29T11:05:13.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.176+0000 7f4363fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:13.155 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.176+0000 7f4363fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:13.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.508+0000 7f437c8ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:13.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.509+0000 7f437c8ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:13.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.509+0000 7f437c8ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.22:6790/0 2024-03-29T11:05:13.488 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.509+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 --> v1:172.21.15.22:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f437002dd30 con 0x7f4370006a70 2024-03-29T11:05:13.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.511+0000 7f4363fff640 1 -- 172.21.15.22:0/1126417872 <== mon.2 v1:172.21.15.22:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7f437002dd30 con 0x7f4370006a70 2024-03-29T11:05:13.490 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.511+0000 7f4363fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:13.491 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.511+0000 7f4363fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:13.491 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:05:13.491 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:13.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.515+0000 7f437c8ac640 10 librados: watch_flush enter 2024-03-29T11:05:13.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.517+0000 7f437c8ac640 10 librados: watch_flush exit 2024-03-29T11:05:13.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.517+0000 7f437c8ac640 20 client.4745.objecter shutdown clearing up homeless session... 2024-03-29T11:05:13.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.517+0000 7f437c8ac640 10 client.4745.objecter successfully canceled tick 2024-03-29T11:05:13.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.517+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 >> v1:172.21.15.22:6816/380221644 conn(0x7f435c06de40 legacy=0x7f435c070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:13.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.517+0000 7f437c8ac640 10 monclient: shutdown 2024-03-29T11:05:13.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.518+0000 7f437c8ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:13.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.518+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 >> v1:172.21.15.22:6790/0 conn(0x7f4370006a70 legacy=0x7f4370160de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:13.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.518+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 shutdown_connections 2024-03-29T11:05:13.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.518+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 >> 172.21.15.22:0/1126417872 conn(0x7f43700058f0 msgr2=0x7f43700338b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:13.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.518+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 shutdown_connections 2024-03-29T11:05:13.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.518+0000 7f437c8ac640 1 -- 172.21.15.22:0/1126417872 wait complete. 2024-03-29T11:05:13.500 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:13.521+0000 7f437c8ac640 1 librados: shutdown 2024-03-29T11:05:13.516 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:20.323 DEBUG:teuthology.orchestra.run.smithi022:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.490+0000 7fa1778c8640 10 monclient: get_monmap_and_config 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.490+0000 7fa1778c8640 10 monclient: build_initial_monmap 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.490+0000 7fa1778c8640 10 monclient: monmap: 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T11:05:20.490895+0000 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T11:05:20.490895+0000 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:20.469 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:05:20.470 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-c 2024-03-29T11:05:20.470 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-b 2024-03-29T11:05:20.470 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:20.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.491+0000 7fa1778c8640 1 Processor -- start 2024-03-29T11:05:20.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.491+0000 7fa1778c8640 1 -- start start 2024-03-29T11:05:20.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.491+0000 7fa1778c8640 10 monclient: init 2024-03-29T11:05:20.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.492+0000 7fa1778c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:20.471 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.492+0000 7fa1778c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 10 monclient(hunting): picked mon.noname-b con 0x7fa168006810 addr v1:172.21.15.22:6790/0 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 10 monclient(hunting): picked mon.noname-a con 0x7fa168006bf0 addr v1:172.21.15.22:6789/0 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa174e63640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa168006bf0 0x7fa1680cae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:55860/0 (socket says 172.21.15.22:55860) 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa174e63640 1 -- 172.21.15.22:0/944992537 learned_addr learned my addr 172.21.15.22:0/944992537 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 10 monclient(hunting): picked mon.noname-c con 0x7fa1680c78f0 addr v1:172.21.15.26:6789/0 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1680c7f30 con 0x7fa168006bf0 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa168036fd0 con 0x7fa1680c78f0 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1680c54f0 con 0x7fa168006810 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:20.474 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.494+0000 7fa1778c8640 10 monclient(hunting): authenticate will time out at 1421.798423s 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220727456 0 0) 0x7fa1680c7f30 con 0x7fa168006bf0 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa15c003520 con 0x7fa168006bf0 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586724002 0 0) 0x7fa1680c54f0 con 0x7fa168006810 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): _negotiate 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 10 monclient(hunting): my global_id is 4748 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.496+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1680c7f30 con 0x7fa168006810 2024-03-29T11:05:20.479 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2890774003 0 0) 0x7fa15c003520 con 0x7fa168006bf0 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 >> v1:172.21.15.22:6790/0 conn(0x7fa168006810 legacy=0x7fa168033620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 >> v1:172.21.15.26:6789/0 conn(0x7fa1680c78f0 legacy=0x7fa1680d31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 10 monclient: _finish_hunting 0 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 1 monclient: found mon.noname-a 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.22:6789/0 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa168037240 con 0x7fa168006bf0 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 10 monclient: _finish_auth 0 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 10 monclient: _check_auth_tickets 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa160001690 con 0x7fa168006bf0 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.497+0000 7fa167fff640 10 monclient: dump: 2024-03-29T11:05:20.480 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa1778c8640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa1778c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 <== mon.0 v1:172.21.15.22:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa1600018b0 con 0x7fa168006bf0 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa167fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:20.481 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa167fff640 1 -- 172.21.15.22:0/944992537 <== mon.0 v1:172.21.15.22:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa160001bc0 con 0x7fa168006bf0 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa167fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa167fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa167fff640 10 monclient: dump: 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:20.482 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa1778c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:05:20.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.498+0000 7fa1778c8640 4 set_mon_vals no callback set 2024-03-29T11:05:20.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.499+0000 7fa1778c8640 10 monclient: shutdown 2024-03-29T11:05:20.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.499+0000 7fa1778c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:20.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.499+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 >> v1:172.21.15.22:6789/0 conn(0x7fa168006bf0 legacy=0x7fa1680cae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:20.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.499+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 shutdown_connections 2024-03-29T11:05:20.483 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.499+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 >> 172.21.15.22:0/944992537 conn(0x7fa168005c40 msgr2=0x7fa168006050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.500+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 shutdown_connections 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.501+0000 7fa1778c8640 1 -- 172.21.15.22:0/944992537 wait complete. 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.502+0000 7fa1778c8640 10 monclient: build_initial_monmap 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.502+0000 7fa1778c8640 10 monclient: monmap: 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:epoch 0 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:last_changed 0.000000 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:created 0.000000 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 0 (unknown) 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.noname-a 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.noname-b 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.noname-c 2024-03-29T11:05:20.484 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.502+0000 7fa1778c8640 1 librados: starting msgr at 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.502+0000 7fa1778c8640 1 librados: starting objecter 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.502+0000 7fa1778c8640 1 Processor -- start 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.503+0000 7fa1778c8640 1 -- start start 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.503+0000 7fa1778c8640 1 librados: setting wanted keys 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.503+0000 7fa1778c8640 1 librados: calling monclient init 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.503+0000 7fa1778c8640 10 monclient: init 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.503+0000 7fa1778c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:05:20.485 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.503+0000 7fa1778c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:05:20.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.505+0000 7fa1778c8640 10 monclient(hunting): picked mon.noname-c con 0x7fa168006810 addr v1:172.21.15.22:6790/0 2024-03-29T11:05:20.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa1778c8640 10 monclient(hunting): picked mon.noname-a con 0x7fa1680c78f0 addr v1:172.21.15.22:6789/0 2024-03-29T11:05:20.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa1778c8640 10 monclient(hunting): picked mon.noname-b con 0x7fa1680c3900 addr v1:172.21.15.26:6789/0 2024-03-29T11:05:20.486 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa174e63640 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa1680c78f0 0x7fa16816c9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:56400/0 (socket says 172.21.15.22:56400) 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa174e63640 1 -- 172.21.15.22:0/2271408555 learned_addr learned my addr 172.21.15.22:0/2271408555 (peer_addr_for_me v1:172.21.15.22:0/0) 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16802dcc0 con 0x7fa1680c78f0 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.26:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1680388a0 con 0x7fa1680c3900 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa16802def0 con 0x7fa168006810 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa165ffb640 10 client.?.objecter ms_handle_connect 0x7fa1680c78f0 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa1778c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa165ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:20.487 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.506+0000 7fa1778c8640 10 monclient(hunting): authenticate will time out at 1421.810597s 2024-03-29T11:05:20.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 10 client.?.objecter ms_handle_connect 0x7fa168006810 2024-03-29T11:05:20.492 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.2 v1:172.21.15.22:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329124513 0 0) 0x7fa16802def0 con 0x7fa168006810 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 10 monclient(hunting): my global_id is 4751 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.508+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa154004eb0 con 0x7fa168006810 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514725924 0 0) 0x7fa16802dcc0 con 0x7fa1680c78f0 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): my global_id is 4563 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa16802def0 con 0x7fa1680c78f0 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3261622990 0 0) 0x7fa16802def0 con 0x7fa1680c78f0 2024-03-29T11:05:20.493 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa16802dcc0 con 0x7fa1680c78f0 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa160001610 con 0x7fa1680c78f0 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): dump: 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:20.494 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.509+0000 7fa165ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 894317048 0 0) 0x7fa16802dcc0 con 0x7fa1680c78f0 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 >> v1:172.21.15.22:6790/0 conn(0x7fa168006810 legacy=0x7fa1680c31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 >> v1:172.21.15.26:6789/0 conn(0x7fa1680c3900 legacy=0x7fa16816d0d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 1 monclient: found mon.a 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16802e1d0 con 0x7fa1680c78f0 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 10 monclient: _finish_auth 0 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa165ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa1778c8640 5 monclient: authenticate success, global_id 4563 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa1778c8640 10 monclient: _renew_subs 2024-03-29T11:05:20.495 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa1778c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.510+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa16802d1b0 con 0x7fa1680c78f0 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa1778c8640 10 client.4563.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa1778c8640 10 monclient: _renew_subs 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa1778c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1680cab20 con 0x7fa1680c78f0 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa1778c8640 1 librados: init done 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa160004a90 con 0x7fa1680c78f0 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa165ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 649107712 0 0) 0x7fa160004da0 con 0x7fa1680c78f0 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa165ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa165ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa165ffb640 10 monclient: dump: 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:epoch 1 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:fsid 9704968d-8e2e-420e-abdf-0499c0fc035c 2024-03-29T11:05:20.496 INFO:teuthology.misc.health.smithi022.stderr:last_changed 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:created 2024-03-29T10:53:48.204207+0000 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:min_mon_release 19 (squid) 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:election_strategy: 1 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:0: v1:172.21.15.22:6789/0 mon.a 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:1: v1:172.21.15.26:6789/0 mon.b 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2: v1:172.21.15.22:6790/0 mon.c 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr: 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.511+0000 7fa1677fe640 4 set_mon_vals no callback set 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.513+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 277816957 0 0) 0x7fa16001ad70 con 0x7fa1680c78f0 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.513+0000 7fa1778c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.513+0000 7fa1778c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.513+0000 7fa1778c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.513+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa150005080 con 0x7fa1680c78f0 2024-03-29T11:05:20.497 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.514+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 4119832651 0 0) 0x7fa160084bd0 con 0x7fa1680c78f0 2024-03-29T11:05:20.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.514+0000 7fa165ffb640 10 client.4563.objecter ms_dispatch 0x7fa170000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:05:20.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.514+0000 7fa165ffb640 3 client.4563.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:05:20.498 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.514+0000 7fa165ffb640 3 client.4563.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:05:20.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.514+0000 7fa165ffb640 20 client.4563.objecter dump_active .. 0 homeless 2024-03-29T11:05:20.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.517+0000 7fa175664640 10 monclient: get_auth_request con 0x7fa15406dce0 auth_method 0 2024-03-29T11:05:20.499 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.517+0000 7fa165ffb640 10 client.4563.objecter ms_handle_connect 0x7fa15406dce0 2024-03-29T11:05:20.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.522+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa1600504b0 con 0x7fa1680c78f0 2024-03-29T11:05:20.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.522+0000 7fa165ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:05:20.501 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.522+0000 7fa165ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:05:20.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.855+0000 7fa1778c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:20.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.855+0000 7fa1778c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:20.834 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.855+0000 7fa1778c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.22:6789/0 2024-03-29T11:05:20.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.855+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 --> v1:172.21.15.22:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa150005330 con 0x7fa1680c78f0 2024-03-29T11:05:20.835 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.856+0000 7fa165ffb640 1 -- 172.21.15.22:0/2271408555 <== mon.0 v1:172.21.15.22:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 2428452928) 0x7fa1600523e0 con 0x7fa1680c78f0 2024-03-29T11:05:20.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.856+0000 7fa165ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:05:20.836 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.856+0000 7fa165ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:05:20.836 INFO:teuthology.misc.health.smithi022.stdout: 2024-03-29T11:05:20.836 INFO:teuthology.misc.health.smithi022.stdout:{"epoch":11,"fsid":"9704968d-8e2e-420e-abdf-0499c0fc035c","created":"2024-03-29T10:54:22.502946+0000","modified":"2024-03-29T11:04:27.408826+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:04:27.408826+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"dd397594-0f34-4d00-96b5-a1f909d5cc8c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"89d98e1f-fd1d-4578-bbe3-8e197702debd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"32f00e39-6735-4e83-85f9-49fbd82a22b2","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"30bbbc18-5aa5-4e64-bb3f-192e4c68fae9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0d705184-b655-464a-b001-22221eac4ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"04210bd3-557d-4820-ba3f-2ff1a251cdaa","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"608227e6-bd7d-40be-bfe3-d7205697aef8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"b1d4ed43-f342-4655-91a5-33de40a75723","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:05:20.838 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.859+0000 7fa1778c8640 10 librados: watch_flush enter 2024-03-29T11:05:20.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 10 librados: watch_flush exit 2024-03-29T11:05:20.841 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 20 client.4563.objecter shutdown clearing up homeless session... 2024-03-29T11:05:20.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 10 client.4563.objecter successfully canceled tick 2024-03-29T11:05:20.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 >> v1:172.21.15.22:6816/380221644 conn(0x7fa15406dce0 legacy=0x7fa1540701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:20.842 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 10 monclient: shutdown 2024-03-29T11:05:20.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:05:20.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 >> v1:172.21.15.22:6789/0 conn(0x7fa1680c78f0 legacy=0x7fa16816c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:05:20.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 shutdown_connections 2024-03-29T11:05:20.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 >> 172.21.15.22:0/2271408555 conn(0x7fa168005c40 msgr2=0x7fa1680cb5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:05:20.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 shutdown_connections 2024-03-29T11:05:20.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.863+0000 7fa1778c8640 1 -- 172.21.15.22:0/2271408555 wait complete. 2024-03-29T11:05:20.843 INFO:teuthology.misc.health.smithi022.stderr:2024-03-29T11:05:20.864+0000 7fa1778c8640 1 librados: shutdown 2024-03-29T11:05:20.859 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:05:20.859 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:05:20.861 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:05:20.861 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:05:20.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:20.893 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:05:20.893 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:05:20.893 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:20.972 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:05:20.972 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:05:20.972 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.041 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:05:21.041 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:05:21.041 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.118 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:05:21.118 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:05:21.118 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.162 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:05:21.162 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:05:21.162 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.199 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:05:21.199 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:05:21.199 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.275 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:05:21.275 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:05:21.275 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.346 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:05:21.346 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:05:21.346 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:05:21.346 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.371 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:05:21.372 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:05:21.372 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.410 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:05:21.411 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:05:21.411 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:05:21.411 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.423 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:05:21.423 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:05:21.423 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.475 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:05:21.475 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:05:21.475 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:05:21.487 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:05:21.487 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:05:21.487 DEBUG:teuthology.orchestra.run.smithi022:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:05:21.689 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:05:21.690 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:05:25.856 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:05:25.856 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:05:25.961 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:05:25.961 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:05:26.002 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:05:26.002 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:05:26.084 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T11:05:26.084 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:05:28.996 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T11:05:28.996 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:05:29.090 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T11:05:29.090 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:05:29.149 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T11:05:29.149 DEBUG:teuthology.orchestra.run.smithi026:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:05:29.207 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:05:29.209 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/data/mon.a.tgz 2024-03-29T11:05:29.209 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-29T11:05:29.227 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.gakNJOJP3A 2024-03-29T11:05:29.227 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.gakNJOJP3A 2024-03-29T11:05:29.330 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.gakNJOJP3A 2024-03-29T11:05:29.417 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.gakNJOJP3A is 126KB 2024-03-29T11:05:29.469 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.gakNJOJP3A 2024-03-29T11:05:29.484 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/data/mon.c.tgz 2024-03-29T11:05:29.484 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2024-03-29T11:05:29.539 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.u1Whb2Xkdg 2024-03-29T11:05:29.539 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.u1Whb2Xkdg 2024-03-29T11:05:29.627 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.u1Whb2Xkdg 2024-03-29T11:05:29.710 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.u1Whb2Xkdg is 126KB 2024-03-29T11:05:29.765 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.u1Whb2Xkdg 2024-03-29T11:05:29.779 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/data/mon.b.tgz 2024-03-29T11:05:29.779 DEBUG:teuthology.orchestra.run.smithi026:> mktemp 2024-03-29T11:05:29.797 INFO:teuthology.orchestra.run.smithi026.stdout:/tmp/tmp.LD2OrXe1oz 2024-03-29T11:05:29.797 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.LD2OrXe1oz 2024-03-29T11:05:29.897 DEBUG:teuthology.orchestra.run.smithi026:> sudo chmod 0666 /tmp/tmp.LD2OrXe1oz 2024-03-29T11:05:29.984 DEBUG:teuthology.orchestra.remote:smithi026:/tmp/tmp.LD2OrXe1oz is 126KB 2024-03-29T11:05:30.035 DEBUG:teuthology.orchestra.run.smithi026:> rm -fr /tmp/tmp.LD2OrXe1oz 2024-03-29T11:05:30.049 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:05:30.050 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:05:30.052 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:05:30.132 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:05:30.171 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:05:30.171 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:05:30.173 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/remote/smithi022/crash 2024-03-29T11:05:30.174 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:05:30.201 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/remote/smithi026/crash 2024-03-29T11:05:30.202 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:05:30.237 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:05:30.237 DEBUG:teuthology.orchestra.run.smithi022:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:05:30.246 DEBUG:teuthology.orchestra.run.smithi026:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:05:30.311 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39767.log 2024-03-29T11:05:30.311 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:05:30.311 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39650.log 2024-03-29T11:05:30.311 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39767.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34545.log 2024-03-29T11:05:30.311 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39650.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40920.log 2024-03-29T11:05:30.311 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39767.log.gz 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.34545.log: gzip -5 86.6% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.39650.log.gz /var/log/ceph/ceph-client.admin.40803.log 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40920.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34545.log.gz 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40959.log 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40370.log 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40920.log.gz 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40959.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40803.log.gz 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40198.log 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40959.log.gz 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40370.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40685.log 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40998.log 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40370.log.gz 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40685.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40198.log.gz 2024-03-29T11:05:30.312 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41232.log 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40487.log 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40685.log.gz 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41232.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39533.log 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40998.log.gz 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40487.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41232.log.gz 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40646.log 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40042.log 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39533.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40487.log.gz 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40646.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41700.log 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39533.log.gz 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40646.log.gz 2024-03-29T11:05:30.313 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39845.log 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39143.log 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41700.log: /var/log/ceph/ceph-client.admin.39845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38753.log 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40042.log.gz 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41700.log.gz 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41037.log 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39845.log.gz 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38753.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39806.log 2024-03-29T11:05:30.314 INFO:teuthology.orchestra.run.smithi022.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39143.log.gz 2024-03-29T11:05:30.315 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41466.log 2024-03-29T11:05:30.315 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38753.log.gz 2024-03-29T11:05:30.315 INFO:teuthology.orchestra.run.smithi022.stderr: 86.0%/var/log/ceph/ceph-client.admin.39806.log: -- replaced with /var/log/ceph/ceph-client.admin.41037.log.gz 2024-03-29T11:05:30.315 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39221.log 2024-03-29T11:05:30.315 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41466.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38322.log 2024-03-29T11:05:30.315 INFO:teuthology.orchestra.run.smithi022.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39806.log.gz 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39221.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41466.log.gz 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40724.log 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38870.log 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39221.log.gz 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41661.log 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38322.log.gz 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38870.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40724.log.gz 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39728.log 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41622.log 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38870.log.gz 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41505.log 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41661.log.gz 2024-03-29T11:05:30.316 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39416.log 86.1% 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39728.log.gz 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41505.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41622.log.gz 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41505.log.gz/var/log/ceph/ceph-client.admin.39416.log: 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38597.log 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41544.log 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39416.log.gz 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39455.log 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.38597.log.gz 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39494.log 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41544.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41583.log 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39455.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41544.log.gz 2024-03-29T11:05:30.317 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39572.log 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39455.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39494.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40159.log 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41583.log: /var/log/ceph/ceph-client.admin.39572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38244.log 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39884.log 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41583.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39572.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39182.log 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40159.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39260.log 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38244.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39182.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39884.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40764.log 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39260.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39182.log.gz 2024-03-29T11:05:30.318 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39689.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40764.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39260.log.gz 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39689.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41154.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40764.log.gz 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28714.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39689.log.gz 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41271.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.tmp-client.admin.28714.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41154.log.gz 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28714.log.gz 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40003.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38948.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41271.log: /var/log/ceph/ceph-client.admin.40003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39338.log 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38948.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41271.log.gz 2024-03-29T11:05:30.319 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40120.log 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40003.log.gz 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39377.log 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.38948.log.gz 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41193.log 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39338.log.gz 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5/var/log/ceph/ceph-client.admin.39377.log: --verbose -- /var/log/ceph/ceph-client.admin.39962.log 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40120.log.gz 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41193.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39299.log 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39377.log.gz 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39962.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41193.log.gz 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39962.log.gz 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38400.log 2024-03-29T11:05:30.320 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38909.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.3.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39299.log.gz 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38831.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38909.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38400.log.gz 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34939.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41427.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38831.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38909.log.gz 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.34939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38519.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38831.log.gz 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38714.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34939.log.gz 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:gzip/var/log/ceph/ceph-client.admin.38519.log: -5 --verbose -- 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41427.log.gz 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr: /var/log/ceph/ceph-client.admin.40237.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39065.log 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38519.log.gz 2024-03-29T11:05:30.321 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40237.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38714.log.gz 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34836.log 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41310.log 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39065.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40237.log.gz 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.34836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40881.log 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39065.log.gz 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40409.log 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.34836.log.gz 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40881.log: gzip -5 --verbose -- 85.3% -- replaced with /var/log/ceph/ceph-client.admin.41310.log.gz 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr: /var/log/ceph/ceph-client.admin.40276.log 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40409.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40881.log.gz 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40409.log.gz 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38478.log 2024-03-29T11:05:30.322 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41076.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-mgr.y.log: 85.9%/var/log/ceph/ceph-client.admin.38478.log: -- replaced with /var/log/ceph/ceph-client.admin.40276.log.gz 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38439.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41076.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38478.log.gz 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39923.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:gzip/var/log/ceph/ceph-client.admin.38439.log: -5 --verbose 85.9% -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41076.log.gz 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41115.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38439.log.gz 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39923.log.gz 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.0.log: --verbose -- /var/log/ceph/ceph-client.admin.39611.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38792.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39611.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41115.log.gz 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39026.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39611.log.gz 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38792.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38987.log 2024-03-29T11:05:30.323 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39026.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr: 86.9%/var/log/ceph/ceph-client.admin.38987.log: -- replaced with /var/log/ceph/ceph-client.admin.38792.log.gz 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39026.log.gz 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38283.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38987.log.gz 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40568.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.38283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40448.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40568.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.38283.log.gz 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40568.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40448.log: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40448.log.gz 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39104.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38558.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.39104.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40081.log 2024-03-29T11:05:30.324 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38558.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39104.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41349.log 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr: 86.1%/var/log/ceph/ceph-client.admin.40081.log: -- replaced with /var/log/ceph/ceph-client.admin.38558.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40081.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40526.log 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38361.log 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40526.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41349.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40331.log 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38361.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40526.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38361.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40331.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41388.log 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40607.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40331.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38675.log 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-03-29T11:05:30.325 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.41388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38636.log 2024-03-29T11:05:30.326 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38675.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41388.log.gz 2024-03-29T11:05:30.326 INFO:teuthology.orchestra.run.smithi022.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38675.log.gz 2024-03-29T11:05:30.326 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:05:30.326 INFO:teuthology.orchestra.run.smithi022.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40842.log 2024-03-29T11:05:30.326 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.38636.log: /var/log/ceph/ceph-mon.c.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.38636.log.gz 2024-03-29T11:05:30.326 INFO:teuthology.orchestra.run.smithi022.stderr:/var/log/ceph/ceph-client.admin.40842.log: 85.4% -- replaced with /var/log/ceph/ceph-client.admin.40842.log.gz 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34278.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34518.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34278.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34518.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34278.log.gz 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph.audit.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34518.log.gz 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34398.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34358.log 2024-03-29T11:05:30.332 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34478.log 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34358.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34398.log.gz 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34478.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.34358.log.gzgzip 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr: -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr: 87.8%/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.34478.log.gz/var/log/ceph/ceph-mgr.x.log: 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34438.log 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34238.log 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34438.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34438.log.gz 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34318.log 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr:/var/log/ceph/ceph-client.admin.34238.log: /var/log/ceph/ceph-client.admin.34318.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34238.log.gz 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34318.log.gz 2024-03-29T11:05:30.333 INFO:teuthology.orchestra.run.smithi026.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:05:30.366 INFO:teuthology.orchestra.run.smithi022.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:05:42.675 INFO:teuthology.orchestra.run.smithi022.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:05:42.703 INFO:teuthology.orchestra.run.smithi026.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:05:42.733 INFO:teuthology.orchestra.run.smithi026.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:05:42.839 INFO:teuthology.orchestra.run.smithi026.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:05:45.664 INFO:teuthology.orchestra.run.smithi022.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:05:46.821 INFO:teuthology.orchestra.run.smithi022.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:05:46.871 INFO:teuthology.orchestra.run.smithi022.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:05:47.107 INFO:teuthology.orchestra.run.smithi026.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:06:18.518 INFO:teuthology.orchestra.run.smithi022.stderr: 92.9% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:06:33.331 INFO:teuthology.orchestra.run.smithi022.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:06:33.333 INFO:teuthology.orchestra.run.smithi022.stderr: 2024-03-29T11:06:33.333 INFO:teuthology.orchestra.run.smithi022.stderr:real 1m3.075s 2024-03-29T11:06:33.333 INFO:teuthology.orchestra.run.smithi022.stderr:user 2m46.380s 2024-03-29T11:06:33.333 INFO:teuthology.orchestra.run.smithi022.stderr:sys 0m5.456s 2024-03-29T11:06:41.272 INFO:teuthology.orchestra.run.smithi026.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:06:41.274 INFO:teuthology.orchestra.run.smithi026.stderr: 2024-03-29T11:06:41.274 INFO:teuthology.orchestra.run.smithi026.stderr:real 1m10.978s 2024-03-29T11:06:41.274 INFO:teuthology.orchestra.run.smithi026.stderr:user 2m0.796s 2024-03-29T11:06:41.274 INFO:teuthology.orchestra.run.smithi026.stderr:sys 0m4.110s 2024-03-29T11:06:41.274 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:06:41.275 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/remote/smithi022/log 2024-03-29T11:06:41.276 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:07:03.777 DEBUG:teuthology.misc:Transferring archived files from smithi026:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/remote/smithi026/log 2024-03-29T11:07:03.778 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:07:21.139 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:07:21.350 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a733d30ad46d4fd098f2d4c549eb7d8e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:07:21.354 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:07:21.369 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:07:21.382 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:07:21.382 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:07:21.396 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:07:21.475 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:07:21.486 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:07:21.487 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:07:21.490 DEBUG:teuthology.orchestra.run.smithi026:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:07:21.504 INFO:teuthology.orchestra.run.smithi022.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:07:21.506 INFO:teuthology.orchestra.run.smithi026.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:07:21.508 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:07:21.508 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2024-03-29T11:07:21.508 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 47 +408us[ +408us] +/- 69ms 2024-03-29T11:07:21.508 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 51 -214us[ -238us] +/- 64ms 2024-03-29T11:07:21.508 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -189us[ -189us] +/- 63ms 2024-03-29T11:07:21.508 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:07:21.513 INFO:teuthology.orchestra.run.smithi026.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:07:21.513 INFO:teuthology.orchestra.run.smithi026.stdout:=============================================================================== 2024-03-29T11:07:21.514 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 +414us[ +410us] +/- 69ms 2024-03-29T11:07:21.514 INFO:teuthology.orchestra.run.smithi026.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -189us[ -193us] +/- 64ms 2024-03-29T11:07:21.514 INFO:teuthology.orchestra.run.smithi026.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 51 -201us[ -205us] +/- 63ms 2024-03-29T11:07:21.514 INFO:teuthology.orchestra.run.smithi026.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:07:21.515 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:07:21.524 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:07:21.524 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:07:21.537 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:07:21.621 DEBUG:teuthology.orchestra.run.smithi026:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:07:21.692 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:07:21.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:21.725 DEBUG:teuthology.orchestra.run.smithi026:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:07:21.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:21.757 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:07:21.767 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:07:21.780 INFO:teuthology.task.internal:Duration was 1224.384472 seconds 2024-03-29T11:07:21.780 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:07:21.794 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:07:21.795 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:07:21.797 DEBUG:teuthology.orchestra.run.smithi026:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:07:21.841 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:07:21.843 INFO:teuthology.orchestra.run.smithi026.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:07:22.146 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:07:22.146 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi022.front.sepia.ceph.com 2024-03-29T11:07:22.147 DEBUG:teuthology.orchestra.run.smithi022:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:07:22.167 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi026.front.sepia.ceph.com 2024-03-29T11:07:22.168 DEBUG:teuthology.orchestra.run.smithi026:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:07:22.193 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:07:22.193 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:07:22.210 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:07:22.263 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:07:22.263 DEBUG:teuthology.orchestra.run.smithi022:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:07:22.280 DEBUG:teuthology.orchestra.run.smithi026:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:07:22.401 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:07:22.411 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:07:22.412 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:07:22.438 DEBUG:teuthology.orchestra.run.smithi026:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:07:22.464 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:07:22.474 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:07:22.481 DEBUG:teuthology.orchestra.run.smithi026:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:07:22.503 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2024-03-29T11:07:22.530 INFO:teuthology.orchestra.run.smithi026.stdout:kernel.core_pattern = core 2024-03-29T11:07:22.554 DEBUG:teuthology.orchestra.run.smithi022:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:07:22.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:22.579 DEBUG:teuthology.orchestra.run.smithi026:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:07:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:07:22.608 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:07:22.619 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:07:22.621 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/remote/smithi022 2024-03-29T11:07:22.621 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:07:22.657 DEBUG:teuthology.misc:Transferring archived files from smithi026:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663/remote/smithi026 2024-03-29T11:07:22.657 DEBUG:teuthology.orchestra.run.smithi026:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:07:22.691 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:07:22.692 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:07:22.694 DEBUG:teuthology.orchestra.run.smithi026:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:07:22.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:07:22.752 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:07:22.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:07:22.764 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:07:22.764 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:07:22.767 DEBUG:teuthology.orchestra.run.smithi026:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:07:22.778 INFO:teuthology.orchestra.run.smithi022.stdout: 262170 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:07 /home/ubuntu/cephtest 2024-03-29T11:07:22.795 INFO:teuthology.orchestra.run.smithi026.stdout: 265916 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:07 /home/ubuntu/cephtest 2024-03-29T11:07:22.796 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:07:22.808 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:07:22.834 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:07:22.861 DEBUG:teuthology.nuke:shortname: smithi022 2024-03-29T11:07:22.861 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:07:22.884 DEBUG:teuthology.nuke:shortname: smithi026 2024-03-29T11:07:22.885 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:07:22.890 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:39:22.272230', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOyACxNVF/D20cEsz4I+IiCBGhDOKyJdc1ozs7gI/OY9v8Yi188AgmNl4XqE171C391oJoQ0WCjrZbTwX7R5NnY='} 2024-03-29T11:07:22.914 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi026.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622663', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 10:39:22.273777', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGwmYzMU7GD+l5piL0UNy8nu1PbS3hfMr10PIsafCqxd5VMKzXTWrdcyK3b1TFXRWHs5lh+mJVBBao2kK5ilz54='} 2024-03-29T11:07:22.939 INFO:teuthology.orchestra.console.smithi022:Power off 2024-03-29T11:07:22.940 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:07:22.966 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2024-03-29T11:07:22.967 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:07:22.985 INFO:teuthology.orchestra.console.smithi026:Power off 2024-03-29T11:07:22.985 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:07:22.994 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-29T11:07:23.007 DEBUG:teuthology.orchestra.console.smithi026:power off output: Chassis Power Control: Down/Off 2024-03-29T11:07:23.008 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:07:23.031 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-29T11:07:26.996 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:07:27.032 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:07:27.106 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2024-03-29T11:07:27.143 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is on 2024-03-29T11:07:31.112 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:07:31.144 DEBUG:teuthology.orchestra.console.smithi026:pexpect command: ipmitool -H smithi026.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:07:31.223 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2024-03-29T11:07:31.254 DEBUG:teuthology.orchestra.console.smithi026:check power output: Chassis Power is off 2024-03-29T11:07:31.324 INFO:teuthology.orchestra.console.smithi022:Power off completed 2024-03-29T11:07:31.355 INFO:teuthology.orchestra.console.smithi026:Power off completed 2024-03-29T11:07:31.460 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-zstd rados tasks/rados_api_tests validater/lockdep} duration: 1224.384471654892 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a733d30ad46d4fd098f2d4c549eb7d8e status: fail success: false 2024-03-29T11:07:31.460 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:07:31.547 INFO:teuthology.run:FAIL